Jul 30 02:26:04.259447 ip-10-0-132-89 systemd[1]: Starting Kubernetes Kubelet...
Jul 30 02:26:04.576118 ip-10-0-132-89 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 02:26:04.576118 ip-10-0-132-89 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 30 02:26:04.576118 ip-10-0-132-89 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 02:26:04.576118 ip-10-0-132-89 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 30 02:26:04.576118 ip-10-0-132-89 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 02:26:04.576989 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.576850    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 30 02:26:04.580420 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580400    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 02:26:04.580420 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580416    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 02:26:04.580420 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580421    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 02:26:04.580420 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580427    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 02:26:04.580664 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580431    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 02:26:04.580664 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580435    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 02:26:04.580664 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580439    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 02:26:04.580664 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580442    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 02:26:04.580664 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580446    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 02:26:04.580664 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580450    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 02:26:04.580664 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580454    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 02:26:04.580664 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580457    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 02:26:04.580664 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580461    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 02:26:04.580664 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580464    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 02:26:04.580664 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580468    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 02:26:04.580664 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580472    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 02:26:04.580664 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580476    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 02:26:04.580664 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580479    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 02:26:04.580664 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580483    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 02:26:04.580664 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580487    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 02:26:04.580664 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580490    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 02:26:04.580664 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580494    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 02:26:04.580664 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580500    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 02:26:04.580664 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580508    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 02:26:04.581497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580515    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 02:26:04.581497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580519    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 02:26:04.581497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580523    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 02:26:04.581497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580528    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 02:26:04.581497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580532    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 02:26:04.581497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580536    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 02:26:04.581497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580540    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 02:26:04.581497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580544    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 02:26:04.581497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580548    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 02:26:04.581497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580553    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 02:26:04.581497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580558    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 02:26:04.581497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580563    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 02:26:04.581497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580567    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 02:26:04.581497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580571    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 02:26:04.581497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580576    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 02:26:04.581497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580580    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 02:26:04.581497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580584    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 02:26:04.581497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580588    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 02:26:04.581497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580593    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 02:26:04.581497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580598    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 02:26:04.582356 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580602    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 02:26:04.582356 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580606    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 02:26:04.582356 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580611    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 02:26:04.582356 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580615    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 02:26:04.582356 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580619    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 02:26:04.582356 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580622    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 02:26:04.582356 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580627    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 02:26:04.582356 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580630    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 02:26:04.582356 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580635    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 02:26:04.582356 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580639    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 02:26:04.582356 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580643    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 02:26:04.582356 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580648    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 02:26:04.582356 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580652    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 02:26:04.582356 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580656    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 02:26:04.582356 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580660    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 02:26:04.582356 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580664    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 02:26:04.582356 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580670    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 02:26:04.582356 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580676    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 02:26:04.582356 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580680    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 02:26:04.582356 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580684    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 02:26:04.583187 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580688    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 02:26:04.583187 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580693    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 02:26:04.583187 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580697    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 02:26:04.583187 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580701    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 02:26:04.583187 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580705    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 02:26:04.583187 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580712    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 02:26:04.583187 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580717    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 02:26:04.583187 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580723    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 02:26:04.583187 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580728    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 02:26:04.583187 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580733    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 02:26:04.583187 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580738    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 02:26:04.583187 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580742    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 02:26:04.583187 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580746    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 02:26:04.583187 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580751    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 02:26:04.583187 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580755    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 02:26:04.583187 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580759    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 02:26:04.583187 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580763    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 02:26:04.583187 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580767    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 02:26:04.583187 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580772    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 02:26:04.584128 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580776    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 02:26:04.584128 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580781    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 02:26:04.584128 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580785    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 02:26:04.584128 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580789    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 02:26:04.584128 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580793    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 02:26:04.584128 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580797    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 02:26:04.584128 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580801    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 02:26:04.584128 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580806    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 02:26:04.584128 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580810    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 02:26:04.584128 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580814    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 02:26:04.584128 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580819    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 02:26:04.584128 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580824    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 02:26:04.584128 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580828    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 02:26:04.584128 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580832    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 02:26:04.584128 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580837    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 02:26:04.584128 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.580841    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 02:26:04.584128 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582037    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 02:26:04.584128 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582046    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 02:26:04.584128 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582051    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 02:26:04.584128 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582055    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 02:26:04.584994 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582060    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 02:26:04.584994 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582065    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 02:26:04.584994 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582070    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 02:26:04.584994 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582075    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 02:26:04.584994 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582079    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 02:26:04.584994 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582083    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 02:26:04.584994 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582087    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 02:26:04.584994 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582092    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 02:26:04.584994 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582096    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 02:26:04.584994 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582100    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 02:26:04.584994 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582104    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 02:26:04.584994 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582108    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 02:26:04.584994 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582112    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 02:26:04.584994 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582116    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 02:26:04.584994 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582123    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 02:26:04.584994 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582131    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 02:26:04.584994 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582136    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 02:26:04.584994 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582142    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 02:26:04.584994 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582146    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 02:26:04.585561 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582150    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 02:26:04.585561 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582155    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 02:26:04.585561 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582159    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 02:26:04.585561 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582183    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 02:26:04.585561 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582188    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 02:26:04.585561 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582191    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 02:26:04.585561 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582195    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 02:26:04.585561 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582199    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 02:26:04.585561 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582203    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 02:26:04.585561 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582208    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 02:26:04.585561 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582212    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 02:26:04.585561 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582216    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 02:26:04.585561 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582220    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 02:26:04.585561 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582224    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 02:26:04.585561 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582228    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 02:26:04.585561 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582233    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 02:26:04.585561 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582237    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 02:26:04.585561 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582241    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 02:26:04.585561 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582245    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 02:26:04.585561 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582251    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 02:26:04.586122 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582255    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 02:26:04.586122 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582259    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 02:26:04.586122 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582263    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 02:26:04.586122 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582267    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 02:26:04.586122 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582271    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 02:26:04.586122 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582275    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 02:26:04.586122 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582280    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 02:26:04.586122 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582284    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 02:26:04.586122 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582288    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 02:26:04.586122 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582292    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 02:26:04.586122 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582296    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 02:26:04.586122 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582300    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 02:26:04.586122 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582306    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 02:26:04.586122 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582313    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 02:26:04.586122 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582317    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 02:26:04.586122 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582321    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 02:26:04.586122 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582325    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 02:26:04.586122 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582329    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 02:26:04.586122 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582333    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 02:26:04.586122 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582337    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 02:26:04.586868 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582341    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 02:26:04.586868 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582345    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 02:26:04.586868 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582349    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 02:26:04.586868 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582353    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 02:26:04.586868 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582357    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 02:26:04.586868 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582361    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 02:26:04.586868 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582365    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 02:26:04.586868 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582368    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 02:26:04.586868 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582374    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 02:26:04.586868 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582378    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 02:26:04.586868 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582382    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 02:26:04.586868 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582388    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 02:26:04.586868 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582392    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 02:26:04.586868 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582397    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 02:26:04.586868 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582401    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 02:26:04.586868 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582405    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 02:26:04.586868 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582410    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 02:26:04.586868 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582414    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 02:26:04.586868 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582418    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 02:26:04.586868 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582422    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 02:26:04.587406 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582426    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 02:26:04.587406 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582430    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 02:26:04.587406 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582434    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 02:26:04.587406 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582439    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 02:26:04.587406 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582443    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 02:26:04.587406 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582448    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 02:26:04.587406 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582452    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 02:26:04.587406 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582457    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 02:26:04.587406 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582461    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 02:26:04.587406 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582465    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 02:26:04.587406 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582469    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 02:26:04.587406 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582474    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 02:26:04.587406 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582478    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 02:26:04.587406 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582482    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 02:26:04.587406 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582486    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 02:26:04.587406 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.582490    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 02:26:04.587406 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583133    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 02:26:04.587406 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583143    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 02:26:04.587406 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583148    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 02:26:04.587406 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583153    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 02:26:04.587942 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583158    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 02:26:04.587942 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583182    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 02:26:04.587942 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583187    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 02:26:04.587942 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583192    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 02:26:04.587942 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583196    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 02:26:04.587942 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583200    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 02:26:04.587942 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583204    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 02:26:04.587942 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583208    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 02:26:04.587942 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583213    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 02:26:04.587942 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583217    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 02:26:04.587942 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583221    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 02:26:04.587942 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583224    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 02:26:04.587942 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583229    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 02:26:04.587942 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583234    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 02:26:04.587942 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583238    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 02:26:04.587942 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583243    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 02:26:04.587942 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583247    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 02:26:04.587942 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583254    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 02:26:04.587942 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583258    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 02:26:04.587942 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583262    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 02:26:04.588513 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583266    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 02:26:04.588513 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583270    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 02:26:04.588513 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583274    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 02:26:04.588513 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583278    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 02:26:04.588513 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583282    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 02:26:04.588513 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583286    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 02:26:04.588513 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583291    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 02:26:04.588513 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583295    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 02:26:04.588513 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583299    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 02:26:04.588513 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583303    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 02:26:04.588513 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583307    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 02:26:04.588513 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583311    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 02:26:04.588513 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583315    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 02:26:04.588513 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583321    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 02:26:04.588513 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583328    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 02:26:04.588513 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583334    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 02:26:04.588513 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583339    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 02:26:04.588513 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583344    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 02:26:04.588513 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583348    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 02:26:04.588513 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583352    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 02:26:04.589020 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583357    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 02:26:04.589020 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583361    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 02:26:04.589020 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583365    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 02:26:04.589020 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583369    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 02:26:04.589020 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583375    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 02:26:04.589020 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583379    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 02:26:04.589020 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583383    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 02:26:04.589020 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583387    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 02:26:04.589020 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583391    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 02:26:04.589020 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583397    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 02:26:04.589020 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583401    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 02:26:04.589020 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583405    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 02:26:04.589020 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583410    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 02:26:04.589020 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583414    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 02:26:04.589020 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583418    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 02:26:04.589020 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583422    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 02:26:04.589020 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583426    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 02:26:04.589020 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583430    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 02:26:04.589020 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583434    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 02:26:04.589476 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583439    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 02:26:04.589476 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583445    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 02:26:04.589476 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583451    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 02:26:04.589476 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583455    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 02:26:04.589476 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583459    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 02:26:04.589476 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583463    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 02:26:04.589476 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583467    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 02:26:04.589476 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583471    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 02:26:04.589476 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583477    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 02:26:04.589476 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583481    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 02:26:04.589476 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583486    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 02:26:04.589476 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583490    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 02:26:04.589476 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583494    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 02:26:04.589476 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583498    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 02:26:04.589476 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583502    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 02:26:04.589476 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583506    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 02:26:04.589476 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583511    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 02:26:04.589476 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583518    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 02:26:04.589476 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583523    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583528    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583532    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583537    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583544    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583550    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583554    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583558    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583562    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583566    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583571    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583575    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583579    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583583    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583587    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583591    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583596    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.583600    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583699    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583709    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583719    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 30 02:26:04.589971 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583726    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583739    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583744    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583750    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583757    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583762    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583767    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583773    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583778    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583783    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583788    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583793    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583798    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583802    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583807    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583815    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583819    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583824    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583830    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583835    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583841    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583846    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583851    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 30 02:26:04.590476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583856    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583861    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583866    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583870    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583875    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583880    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583886    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583892    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583896    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583900    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583906    2565 flags.go:64] FLAG: --enable-server="true"
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583911    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583918    2565 flags.go:64] FLAG: --event-burst="100"
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583923    2565 flags.go:64] FLAG: --event-qps="50"
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583927    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583931    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583935    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583941    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583945    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583950    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583954    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583958    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583962    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583967    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583973    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 30 02:26:04.591114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583977    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583982    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583986    2565 flags.go:64] FLAG: --feature-gates=""
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583991    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.583995    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584000    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584004    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584009    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584013    2565 flags.go:64] FLAG: --help="false"
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584017    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-132-89.ec2.internal"
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584023    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584027    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584032    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584038    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584043    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584048    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584052    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584058    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584063    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584069    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584074    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584079    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584083    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584088    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 30 02:26:04.592225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584093    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584097    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584102    2565 flags.go:64] FLAG: --lock-file=""
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584107    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584112    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584117    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584125    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584131    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584135    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584140    2565 flags.go:64] FLAG: --logging-format="text"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584145    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584150    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584155    2565 flags.go:64] FLAG: --manifest-url=""
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584159    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584187    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584191    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584199    2565 flags.go:64] FLAG: --max-pods="110"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584204    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584208    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584213    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584218    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584223    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584227    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584232    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584245    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 30 02:26:04.593251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584252    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584257    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584262    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584267    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584275    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584282    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584287    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584291    2565 flags.go:64] FLAG: --port="10250"
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584296    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584301    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08a86c13e4b148c8c"
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584306    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584311    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584316    2565 flags.go:64] FLAG: --register-node="true"
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584320    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584327    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584332    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584336    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584341    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584346    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584351    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584356    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584360    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584365    2565 flags.go:64] FLAG: --runonce="false"
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584370    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584375    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 30 02:26:04.594332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584379    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584385    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584389    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584394    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584399    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584404    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584409    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584415    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584420    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584424    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584429    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584434    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584439    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584447    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584452    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584457    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584463    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584468    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584472    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584477    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584482    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584487    2565 flags.go:64] FLAG: --v="2"
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584493    2565 flags.go:64] FLAG: --version="false"
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584500    2565 flags.go:64] FLAG: --vmodule=""
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584506    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 30 02:26:04.595229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.584511    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 30 02:26:04.596206 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584694    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 02:26:04.596206 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584702    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 02:26:04.596206 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584708    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 02:26:04.596206 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584713    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 02:26:04.596206 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584717    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 02:26:04.596206 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584722    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 02:26:04.596206 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584726    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 02:26:04.596206 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584731    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 02:26:04.596206 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584735    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 02:26:04.596206 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584739    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 02:26:04.596206 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584743    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 02:26:04.596206 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584747    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 02:26:04.596206 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584751    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 02:26:04.596206 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584757    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 02:26:04.596206 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584762    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 02:26:04.596206 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584766    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 02:26:04.596206 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584770    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 02:26:04.596206 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584774    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 02:26:04.596206 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584779    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 02:26:04.596206 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584783    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 02:26:04.596945 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584787    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 02:26:04.596945 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584792    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 02:26:04.596945 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584796    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 02:26:04.596945 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584800    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 02:26:04.596945 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584805    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 02:26:04.596945 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584809    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 02:26:04.596945 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584813    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 02:26:04.596945 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584818    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 02:26:04.596945 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584822    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 02:26:04.596945 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584826    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 02:26:04.596945 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584830    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 02:26:04.596945 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584834    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 02:26:04.596945 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584838    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 02:26:04.596945 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584842    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 02:26:04.596945 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584846    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 02:26:04.596945 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584850    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 02:26:04.596945 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584854    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 02:26:04.596945 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584858    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 02:26:04.596945 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584862    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 02:26:04.596945 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584866    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 02:26:04.597609 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584870    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 02:26:04.597609 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584874    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 02:26:04.597609 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584878    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 02:26:04.597609 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584882    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 02:26:04.597609 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584886    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 02:26:04.597609 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584892    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 02:26:04.597609 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584896    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 02:26:04.597609 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584900    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 02:26:04.597609 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584904    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 02:26:04.597609 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584909    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 02:26:04.597609 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584915    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 02:26:04.597609 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584921    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 02:26:04.597609 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584927    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 02:26:04.597609 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584931    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 02:26:04.597609 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584936    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 02:26:04.597609 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584940    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 02:26:04.597609 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584944    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 02:26:04.597609 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584948    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 02:26:04.597609 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584952    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 02:26:04.597609 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584957    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 02:26:04.598123 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584961    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 02:26:04.598123 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584965    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 02:26:04.598123 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584969    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 02:26:04.598123 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584973    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 02:26:04.598123 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584977    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 02:26:04.598123 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584981    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 02:26:04.598123 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584985    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 02:26:04.598123 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584989    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 02:26:04.598123 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584992    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 02:26:04.598123 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.584996    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 02:26:04.598123 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585000    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 02:26:04.598123 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585004    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 02:26:04.598123 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585007    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 02:26:04.598123 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585011    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 02:26:04.598123 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585015    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 02:26:04.598123 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585019    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 02:26:04.598123 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585023    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 02:26:04.598123 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585030    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 02:26:04.598123 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585034    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 02:26:04.598123 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585038    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 02:26:04.598686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585042    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 02:26:04.598686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585046    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 02:26:04.598686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585050    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 02:26:04.598686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585054    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 02:26:04.598686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585060    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 02:26:04.598686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585065    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 02:26:04.598686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585072    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 02:26:04.598686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585077    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 02:26:04.598686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585082    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 02:26:04.598686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585086    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 02:26:04.598686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585091    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 02:26:04.598686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585096    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 02:26:04.598686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585100    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 02:26:04.598686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585104    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 02:26:04.598686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585109    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 02:26:04.598686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585113    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 02:26:04.598686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585118    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 02:26:04.598686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585122    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 02:26:04.598686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.585127    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 02:26:04.599285 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.585721    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 02:26:04.599285 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.593346    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 30 02:26:04.599285 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.593363    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 30 02:26:04.599285 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.593392    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 30 02:26:04.599285 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.593401    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 30 02:26:04.599285 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593489    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 02:26:04.599285 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593496    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 02:26:04.599285 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593503    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 02:26:04.599285 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593508    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 02:26:04.599285 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593513    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 02:26:04.599285 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593518    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 02:26:04.599285 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593523    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 02:26:04.599285 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593528    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 02:26:04.599285 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593535    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 02:26:04.599681 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593540    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 02:26:04.599681 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593545    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 02:26:04.599681 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593550    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 02:26:04.599681 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593554    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 02:26:04.599681 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593559    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 02:26:04.599681 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593564    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 02:26:04.599681 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593568    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 02:26:04.599681 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593572    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 02:26:04.599681 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593576    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 02:26:04.599681 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593580    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 02:26:04.599681 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593584    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 02:26:04.599681 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593588    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 02:26:04.599681 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593593    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 02:26:04.599681 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593597    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 02:26:04.599681 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593601    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 02:26:04.599681 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593607    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 02:26:04.599681 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593613    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 02:26:04.599681 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593617    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 02:26:04.599681 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593621    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 02:26:04.599681 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593625    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 02:26:04.600217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593630    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 02:26:04.600217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593634    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 02:26:04.600217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593638    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 02:26:04.600217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593642    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 02:26:04.600217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593646    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 02:26:04.600217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593651    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 02:26:04.600217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593655    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 02:26:04.600217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593659    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 02:26:04.600217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593663    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 02:26:04.600217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593667    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 02:26:04.600217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593694    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 02:26:04.600217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593699    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 02:26:04.600217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593704    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 02:26:04.600217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593708    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 02:26:04.600217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593712    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 02:26:04.600217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593717    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 02:26:04.600217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593721    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 02:26:04.600217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593725    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 02:26:04.600217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593729    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 02:26:04.600217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593734    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 02:26:04.600913 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593738    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 02:26:04.600913 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593742    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 02:26:04.600913 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593746    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 02:26:04.600913 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593750    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 02:26:04.600913 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593754    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 02:26:04.600913 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593758    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 02:26:04.600913 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593762    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 02:26:04.600913 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593766    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 02:26:04.600913 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593770    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 02:26:04.600913 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593775    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 02:26:04.600913 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593779    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 02:26:04.600913 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593783    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 02:26:04.600913 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593787    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 02:26:04.600913 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593792    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 02:26:04.600913 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593796    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 02:26:04.600913 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593800    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 02:26:04.600913 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593805    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 02:26:04.600913 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593808    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 02:26:04.600913 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593813    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 02:26:04.600913 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593817    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 02:26:04.601686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593821    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 02:26:04.601686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593826    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 02:26:04.601686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593830    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 02:26:04.601686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593834    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 02:26:04.601686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593838    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 02:26:04.601686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593842    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 02:26:04.601686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593846    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 02:26:04.601686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593850    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 02:26:04.601686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593854    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 02:26:04.601686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593858    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 02:26:04.601686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593862    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 02:26:04.601686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593867    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 02:26:04.601686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593871    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 02:26:04.601686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593874    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 02:26:04.601686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593879    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 02:26:04.601686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593883    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 02:26:04.601686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593887    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 02:26:04.601686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593894    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 02:26:04.601686 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593899    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 02:26:04.602411 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593903    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 02:26:04.602411 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593907    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 02:26:04.602411 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593911    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 02:26:04.602411 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593915    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 02:26:04.602411 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593920    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 02:26:04.602411 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593924    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 02:26:04.602411 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593928    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 02:26:04.602411 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593932    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 02:26:04.602411 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593937    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 02:26:04.602411 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593941    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 02:26:04.602411 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.593945    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 02:26:04.602411 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.593953    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 02:26:04.602411 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594129    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 02:26:04.602411 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594137    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 02:26:04.602411 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594143    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 02:26:04.603000 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594148    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 02:26:04.603000 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594153    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 02:26:04.603000 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594157    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 02:26:04.603000 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594161    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 02:26:04.603000 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594186    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 02:26:04.603000 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594190    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 02:26:04.603000 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594195    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 02:26:04.603000 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594199    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 02:26:04.603000 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594204    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 02:26:04.603000 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594208    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 02:26:04.603000 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594212    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 02:26:04.603000 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594216    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 02:26:04.603000 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594220    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 02:26:04.603000 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594224    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 02:26:04.603000 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594228    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 02:26:04.603000 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594232    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 02:26:04.603000 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594236    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 02:26:04.603000 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594240    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 02:26:04.603000 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594245    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 02:26:04.603000 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594249    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 02:26:04.603632 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594253    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 02:26:04.603632 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594260    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 02:26:04.603632 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594265    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 02:26:04.603632 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594269    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 02:26:04.603632 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594274    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 02:26:04.603632 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594280    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 02:26:04.603632 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594284    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 02:26:04.603632 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594287    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 02:26:04.603632 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594291    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 02:26:04.603632 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594295    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 02:26:04.603632 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594300    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 02:26:04.603632 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594304    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 02:26:04.603632 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594307    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 02:26:04.603632 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594311    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 02:26:04.603632 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594315    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 02:26:04.603632 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594319    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 02:26:04.603632 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594323    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 02:26:04.603632 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594328    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 02:26:04.603632 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594334    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 02:26:04.604184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594338    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 02:26:04.604184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594341    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 02:26:04.604184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594345    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 02:26:04.604184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594349    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 02:26:04.604184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594353    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 02:26:04.604184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594357    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 02:26:04.604184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594361    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 02:26:04.604184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594366    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 02:26:04.604184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594370    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 02:26:04.604184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594375    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 02:26:04.604184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594379    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 02:26:04.604184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594383    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 02:26:04.604184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594387    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 02:26:04.604184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594391    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 02:26:04.604184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594395    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 02:26:04.604184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594399    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 02:26:04.604184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594403    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 02:26:04.604184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594407    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 02:26:04.604184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594412    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 02:26:04.604184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594416    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 02:26:04.604689 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594421    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 02:26:04.604689 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594427    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 02:26:04.604689 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594433    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 02:26:04.604689 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594438    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 02:26:04.604689 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594442    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 02:26:04.604689 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594446    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 02:26:04.604689 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594450    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 02:26:04.604689 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594454    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 02:26:04.604689 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594458    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 02:26:04.604689 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594462    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 02:26:04.604689 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594467    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 02:26:04.604689 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594473    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 02:26:04.604689 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594477    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 02:26:04.604689 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594481    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 02:26:04.604689 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594485    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 02:26:04.604689 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594489    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 02:26:04.604689 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594493    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 02:26:04.604689 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594497    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 02:26:04.604689 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594502    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 02:26:04.605183 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594506    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 02:26:04.605183 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594510    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 02:26:04.605183 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594514    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 02:26:04.605183 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594518    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 02:26:04.605183 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594522    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 02:26:04.605183 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594526    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 02:26:04.605183 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594530    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 02:26:04.605183 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594534    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 02:26:04.605183 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594538    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 02:26:04.605183 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594542    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 02:26:04.605183 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594546    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 02:26:04.605183 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594550    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 02:26:04.605183 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594554    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 02:26:04.605183 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594559    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 02:26:04.605183 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594563    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 02:26:04.605183 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594570    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 02:26:04.605183 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594575    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 02:26:04.605183 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:04.594580    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 02:26:04.605632 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.594588    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 02:26:04.605632 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.594739    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 30 02:26:04.605632 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.597281    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 30 02:26:04.605632 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.598408    2565 server.go:1019] "Starting client certificate rotation"
Jul 30 02:26:04.605632 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.598506    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 30 02:26:04.605632 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.598538    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 30 02:26:04.605632 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.602111    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 30 02:26:04.605632 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.604017    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 30 02:26:04.615924 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.615908    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 30 02:26:04.617737 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.617723    2565 log.go:25] "Validated CRI v1 image API"
Jul 30 02:26:04.618431 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.618418    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 30 02:26:04.621065 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.621046    2565 fs.go:135] Filesystem UUIDs: map[5a662f06-e4e5-4c30-bf58-3732d4f5843a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c672d6cc-76f8-43ef-be6b-5ed5795d7cd8:/dev/nvme0n1p3]
Jul 30 02:26:04.621125 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.621065    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 30 02:26:04.625139 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.625024    2565 manager.go:216] Machine: {Timestamp:2026-07-30 02:26:04.62455768 +0000 UTC m=+0.271341076 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100780 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23a445fd89af3dd2519a0ac5fe8b6d SystemUUID:ec23a445-fd89-af3d-d251-9a0ac5fe8b6d BootID:9b2c7b5d-e155-4155-b73a-79c0f3cf04a4 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8b:a1:b8:db:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8b:a1:b8:db:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:cf:d3:e3:82:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 30 02:26:04.625139 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.625129    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 30 02:26:04.625382 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.625214    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 30 02:26:04.626485 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.626470    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 30 02:26:04.628017 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.627993    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 30 02:26:04.628193 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.628022    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-89.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 30 02:26:04.628805 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.628793    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 30 02:26:04.628862 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.628811    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 30 02:26:04.628862 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.628822    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 30 02:26:04.629670 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.629658    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 30 02:26:04.630433 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.630421    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 30 02:26:04.633588 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.633574    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 30 02:26:04.633724 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.633712    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 30 02:26:04.633791 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.633752    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 30 02:26:04.633791 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.633766    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 30 02:26:04.633791 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.633780    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 30 02:26:04.633908 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.633793    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 30 02:26:04.633908 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.633806    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 30 02:26:04.634150 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.634136    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 30 02:26:04.634230 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.634158    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 30 02:26:04.635893 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.635870    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 30 02:26:04.637693 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.637676    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 30 02:26:04.637745 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.637711    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 30 02:26:04.639332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.639319    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 30 02:26:04.639397 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.639336    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 30 02:26:04.639397 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.639342    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 30 02:26:04.639397 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.639348    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 30 02:26:04.639397 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.639355    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 30 02:26:04.639397 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.639361    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 30 02:26:04.639397 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.639368    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 30 02:26:04.639397 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.639374    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 30 02:26:04.639397 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.639389    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 30 02:26:04.639397 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.639395    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 30 02:26:04.639619 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.639406    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 30 02:26:04.639619 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.639412    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 30 02:26:04.640462 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.640449    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 30 02:26:04.640494 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.640465    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 30 02:26:04.643609 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.643596    2565 server.go:1268] "Started kubelet"
Jul 30 02:26:04.644781 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.644674    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 30 02:26:04.644985 ip-10-0-132-89 systemd[1]: Started Kubernetes Kubelet.
Jul 30 02:26:04.645106 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.645021    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-89.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 30 02:26:04.645706 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.645646    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 30 02:26:04.645754 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.645747    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 30 02:26:04.645958 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:04.645860    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 30 02:26:04.646099 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:04.646049    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 30 02:26:04.647758 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.647740    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 30 02:26:04.648518 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.648502    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 30 02:26:04.650341 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:04.650321    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 30 02:26:04.650752 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:04.650025    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-89.ec2.internal.18c6f029ce3f349d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-89.ec2.internal,UID:ip-10-0-132-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-89.ec2.internal,},FirstTimestamp:2026-07-30 02:26:04.643554461 +0000 UTC m=+0.290337857,LastTimestamp:2026-07-30 02:26:04.643554461 +0000 UTC m=+0.290337857,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-89.ec2.internal,}"
Jul 30 02:26:04.654637 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.654622    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 30 02:26:04.654637 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.654625    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 30 02:26:04.654793 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:04.654779    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-89.ec2.internal\" not found"
Jul 30 02:26:04.655198 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.655179    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 30 02:26:04.655277 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.655200    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 30 02:26:04.655277 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.655178    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 30 02:26:04.655362 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.655325    2565 factory.go:55] Registering systemd factory
Jul 30 02:26:04.655362 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.655341    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 30 02:26:04.655470 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.655330    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 30 02:26:04.655470 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.655382    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 30 02:26:04.655741 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.655699    2565 factory.go:153] Registering CRI-O factory
Jul 30 02:26:04.655741 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.655739    2565 factory.go:223] Registration of the crio container factory successfully
Jul 30 02:26:04.655857 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.655841    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 30 02:26:04.655912 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.655869    2565 factory.go:103] Registering Raw factory
Jul 30 02:26:04.655912 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.655896    2565 manager.go:1196] Started watching for new ooms in manager
Jul 30 02:26:04.656304 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.656287    2565 manager.go:349] Starting recovery of all containers
Jul 30 02:26:04.656733 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:04.656544    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 30 02:26:04.656733 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:04.656620    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 30 02:26:04.667127 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.667004    2565 manager.go:354] Recovery completed
Jul 30 02:26:04.671181 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.671156    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 02:26:04.673387 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.673367    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 02:26:04.673444 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.673403    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 02:26:04.673444 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.673413    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasSufficientPID"
Jul 30 02:26:04.673977 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.673963    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 30 02:26:04.673977 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.673976    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 30 02:26:04.674108 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.673984    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 02:26:04.674108 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.673993    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 30 02:26:04.674737 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.674723    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 02:26:04.674803 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.674747    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 02:26:04.674803 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.674758    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasSufficientPID"
Jul 30 02:26:04.676669 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.676657    2565 policy_none.go:49] "None policy: Start"
Jul 30 02:26:04.676712 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.676674    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 30 02:26:04.676712 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.676684    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 30 02:26:04.678634 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.678622    2565 policy_none.go:47] "Start"
Jul 30 02:26:04.684651 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:04.684583    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-89.ec2.internal.18c6f029d006667e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-89.ec2.internal,UID:ip-10-0-132-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-89.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-89.ec2.internal,},FirstTimestamp:2026-07-30 02:26:04.67338611 +0000 UTC m=+0.320169506,LastTimestamp:2026-07-30 02:26:04.67338611 +0000 UTC m=+0.320169506,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-89.ec2.internal,}"
Jul 30 02:26:04.694934 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:04.694869    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-89.ec2.internal.18c6f029d006ba7d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-89.ec2.internal,UID:ip-10-0-132-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-89.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-89.ec2.internal,},FirstTimestamp:2026-07-30 02:26:04.673407613 +0000 UTC m=+0.320191009,LastTimestamp:2026-07-30 02:26:04.673407613 +0000 UTC m=+0.320191009,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-89.ec2.internal,}"
Jul 30 02:26:04.703473 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.703346    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bdfmv"
Jul 30 02:26:04.704029 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:04.703970    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-89.ec2.internal.18c6f029d006dcf8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-89.ec2.internal,UID:ip-10-0-132-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-89.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-89.ec2.internal,},FirstTimestamp:2026-07-30 02:26:04.67341644 +0000 UTC m=+0.320199836,LastTimestamp:2026-07-30 02:26:04.67341644 +0000 UTC m=+0.320199836,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-89.ec2.internal,}"
Jul 30 02:26:04.710318 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.710300    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bdfmv"
Jul 30 02:26:04.711246 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:04.711191    2565 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-89.ec2.internal.18c6f029d006667e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-89.ec2.internal.18c6f029d006667e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-89.ec2.internal,UID:ip-10-0-132-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-89.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-89.ec2.internal,},FirstTimestamp:2026-07-30 02:26:04.67338611 +0000 UTC m=+0.320169506,LastTimestamp:2026-07-30 02:26:04.674736315 +0000 UTC m=+0.321519711,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-89.ec2.internal,}"
Jul 30 02:26:04.713696 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.713685    2565 manager.go:337] "Starting Device Plugin manager"
Jul 30 02:26:04.713767 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:04.713713    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 30 02:26:04.713767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.713722    2565 server.go:87] "Starting device plugin registration server"
Jul 30 02:26:04.713939 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.713925    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 30 02:26:04.714015 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.713940    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 30 02:26:04.714099 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.714086    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 30 02:26:04.714200 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.714187    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 30 02:26:04.714200 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.714200    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 30 02:26:04.714608 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:04.714588    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 30 02:26:04.714660 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:04.714644    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-89.ec2.internal\" not found"
Jul 30 02:26:04.792532 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.792498    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 30 02:26:04.794707 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.793833    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 30 02:26:04.794707 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.793879    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 30 02:26:04.794707 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.793913    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 30 02:26:04.794707 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:04.793956    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 30 02:26:04.796578 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.796552    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 02:26:04.814415 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.814393    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 02:26:04.815202 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.815184    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 02:26:04.815266 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.815214    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 02:26:04.815266 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.815224    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasSufficientPID"
Jul 30 02:26:04.815266 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.815246    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-89.ec2.internal"
Jul 30 02:26:04.825870 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.825856    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-89.ec2.internal"
Jul 30 02:26:04.825930 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:04.825877    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-89.ec2.internal\": node \"ip-10-0-132-89.ec2.internal\" not found"
Jul 30 02:26:04.836316 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:04.836270    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-89.ec2.internal\" not found"
Jul 30 02:26:04.894779 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.894754    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-89.ec2.internal"]
Jul 30 02:26:04.894873 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.894829    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 02:26:04.896191 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.896158    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 02:26:04.896262 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.896204    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 02:26:04.896262 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.896221    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasSufficientPID"
Jul 30 02:26:04.898602 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.898590    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 02:26:04.898740 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.898715    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal"
Jul 30 02:26:04.898786 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.898755    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 02:26:04.899228 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.899211    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 02:26:04.899317 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.899235    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 02:26:04.899317 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.899248    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasSufficientPID"
Jul 30 02:26:04.899317 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.899253    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 02:26:04.899317 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.899272    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 02:26:04.899317 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.899282    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasSufficientPID"
Jul 30 02:26:04.901488 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.901474    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-89.ec2.internal"
Jul 30 02:26:04.901555 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.901497    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 02:26:04.902096 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.902083    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 02:26:04.902141 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.902107    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 02:26:04.902141 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.902116    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeHasSufficientPID"
Jul 30 02:26:04.923730 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:04.923712    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-89.ec2.internal\" not found" node="ip-10-0-132-89.ec2.internal"
Jul 30 02:26:04.928153 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:04.928138    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-89.ec2.internal\" not found" node="ip-10-0-132-89.ec2.internal"
Jul 30 02:26:04.937273 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:04.937257    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-89.ec2.internal\" not found"
Jul 30 02:26:04.957129 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.957108    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/74d948a8b8f6069f9ae71aaf0197e360-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal\" (UID: \"74d948a8b8f6069f9ae71aaf0197e360\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal"
Jul 30 02:26:04.957199 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.957135    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74d948a8b8f6069f9ae71aaf0197e360-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal\" (UID: \"74d948a8b8f6069f9ae71aaf0197e360\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal"
Jul 30 02:26:04.957199 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:04.957152    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2615c8560d8f795d489b67609c7125ee-config\") pod \"kube-apiserver-proxy-ip-10-0-132-89.ec2.internal\" (UID: \"2615c8560d8f795d489b67609c7125ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-89.ec2.internal"
Jul 30 02:26:05.038009 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:05.037986    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-89.ec2.internal\" not found"
Jul 30 02:26:05.057974 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.057952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2615c8560d8f795d489b67609c7125ee-config\") pod \"kube-apiserver-proxy-ip-10-0-132-89.ec2.internal\" (UID: \"2615c8560d8f795d489b67609c7125ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-89.ec2.internal"
Jul 30 02:26:05.058034 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.057977    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/74d948a8b8f6069f9ae71aaf0197e360-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal\" (UID: \"74d948a8b8f6069f9ae71aaf0197e360\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal"
Jul 30 02:26:05.058034 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.057995    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74d948a8b8f6069f9ae71aaf0197e360-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal\" (UID: \"74d948a8b8f6069f9ae71aaf0197e360\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal"
Jul 30 02:26:05.058109 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.058041    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74d948a8b8f6069f9ae71aaf0197e360-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal\" (UID: \"74d948a8b8f6069f9ae71aaf0197e360\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal"
Jul 30 02:26:05.058109 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.058050    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2615c8560d8f795d489b67609c7125ee-config\") pod \"kube-apiserver-proxy-ip-10-0-132-89.ec2.internal\" (UID: \"2615c8560d8f795d489b67609c7125ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-89.ec2.internal"
Jul 30 02:26:05.058109 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.058059    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/74d948a8b8f6069f9ae71aaf0197e360-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal\" (UID: \"74d948a8b8f6069f9ae71aaf0197e360\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal"
Jul 30 02:26:05.138420 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:05.138351    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-89.ec2.internal\" not found"
Jul 30 02:26:05.226897 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.226861    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal"
Jul 30 02:26:05.231193 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.231159    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-89.ec2.internal"
Jul 30 02:26:05.239220 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:05.239200    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-89.ec2.internal\" not found"
Jul 30 02:26:05.339703 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:05.339674    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-89.ec2.internal\" not found"
Jul 30 02:26:05.440221 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:05.440144    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-89.ec2.internal\" not found"
Jul 30 02:26:05.540638 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:05.540611    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-89.ec2.internal\" not found"
Jul 30 02:26:05.548047 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.548031    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 02:26:05.555774 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.555754    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-89.ec2.internal"
Jul 30 02:26:05.565530 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.565512    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 30 02:26:05.567023 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.567011    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal"
Jul 30 02:26:05.576339 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.576321    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 30 02:26:05.598481 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.598463    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 30 02:26:05.598624 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.598605    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 30 02:26:05.598671 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.598605    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 30 02:26:05.634787 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.634760    2565 apiserver.go:52] "Watching apiserver"
Jul 30 02:26:05.651337 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.651318    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 30 02:26:05.652258 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.652237    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-w6w7c","kube-system/global-pull-secret-syncer-fsv7s","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6","openshift-cluster-node-tuning-operator/tuned-gjr85","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal","openshift-network-operator/iptables-alerter-9hzfd","openshift-ovn-kubernetes/ovnkube-node-6rrn8","kube-system/konnectivity-agent-b5trq","kube-system/kube-apiserver-proxy-ip-10-0-132-89.ec2.internal","openshift-dns/node-resolver-bzccb","openshift-image-registry/node-ca-bspvc","openshift-multus/multus-77n5d","openshift-multus/multus-additional-cni-plugins-278lc","openshift-multus/network-metrics-daemon-wphj8"]
Jul 30 02:26:05.654718 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.654703    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:05.654799 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.654781    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 30 02:26:05.654873 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:05.654779    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6w7c" podUID="5edd31cd-4288-4f9f-a1a0-a840455c53f2"
Jul 30 02:26:05.656781 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.656765    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:05.656857 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:05.656837    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsv7s" podUID="30a8139c-c65b-45ea-aa17-645a0e4c634a"
Jul 30 02:26:05.658886 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.658847    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.661067 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.661046    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.661184 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.661069    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/30a8139c-c65b-45ea-aa17-645a0e4c634a-dbus\") pod \"global-pull-secret-syncer-fsv7s\" (UID: \"30a8139c-c65b-45ea-aa17-645a0e4c634a\") " pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:05.661184 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.661097    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret\") pod \"global-pull-secret-syncer-fsv7s\" (UID: \"30a8139c-c65b-45ea-aa17-645a0e4c634a\") " pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:05.661184 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.661126    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcd6r\" (UniqueName: \"kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r\") pod \"network-check-target-w6w7c\" (UID: \"5edd31cd-4288-4f9f-a1a0-a840455c53f2\") " pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:05.661184 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.661148    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/30a8139c-c65b-45ea-aa17-645a0e4c634a-kubelet-config\") pod \"global-pull-secret-syncer-fsv7s\" (UID: \"30a8139c-c65b-45ea-aa17-645a0e4c634a\") " pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:05.661451 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.661438    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q28cn\""
Jul 30 02:26:05.661485 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.661449    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 30 02:26:05.661536 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.661521    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 30 02:26:05.661568 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.661553    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 30 02:26:05.663272 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.663246    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9hzfd"
Jul 30 02:26:05.663465 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.663445    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 30 02:26:05.663560 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.663491    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 30 02:26:05.664204 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.664190    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fp77g\""
Jul 30 02:26:05.665568 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.665552    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.666126 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.666111    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n4mnt\""
Jul 30 02:26:05.666679 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.666664    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 30 02:26:05.666760 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.666674    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 30 02:26:05.666801 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.666771    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 30 02:26:05.666989 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.666975    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 30 02:26:05.667712 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.667698    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9knf7\""
Jul 30 02:26:05.667791 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.667729    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b5trq"
Jul 30 02:26:05.668045 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.668032    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 30 02:26:05.669115 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.669089    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 30 02:26:05.669231 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.669187    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 30 02:26:05.669231 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.669210    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 30 02:26:05.669328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.669232    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 30 02:26:05.669328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.669191    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 30 02:26:05.670086 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.670070    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bzccb"
Jul 30 02:26:05.670501 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.670484    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 30 02:26:05.670568 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.670489    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 30 02:26:05.670568 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.670544    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4gwwr\""
Jul 30 02:26:05.672340 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.672324    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bspvc"
Jul 30 02:26:05.673322 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.672855    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 30 02:26:05.673322 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.673043    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 30 02:26:05.673322 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.673050    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wp249\""
Jul 30 02:26:05.675104 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.674857    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 30 02:26:05.675104 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.674959    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 30 02:26:05.675566 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.675550    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.676408 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.676374    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tlvkt\""
Jul 30 02:26:05.676769 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.676753    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 30 02:26:05.677787 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.677759    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 30 02:26:05.678064 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.678049    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 30 02:26:05.678689 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.678149    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 30 02:26:05.678689 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.678217    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.678689 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.678437    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 30 02:26:05.678689 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.678597    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qnnvm\""
Jul 30 02:26:05.680791 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.680680    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:05.680791 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:05.680744    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:26:05.681102 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.681086    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 30 02:26:05.681194 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.681111    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8cwr6\""
Jul 30 02:26:05.681194 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.681112    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 30 02:26:05.694200 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.694130    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l9qwk"
Jul 30 02:26:05.704219 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.704200    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l9qwk"
Jul 30 02:26:05.715209 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.715150    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-29 02:21:04 +0000 UTC" deadline="2028-03-17 06:52:27.303156653 +0000 UTC"
Jul 30 02:26:05.715209 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.715204    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14308h26m21.58795512s"
Jul 30 02:26:05.756277 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.756258    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 30 02:26:05.761346 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/228a0670-c32b-4bf3-a593-4e900bae075b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.761452 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761352    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/228a0670-c32b-4bf3-a593-4e900bae075b-registration-dir\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.761452 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761370    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-os-release\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.761452 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761402    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-kubernetes\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.761612 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761446    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-sys\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.761612 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761484    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-slash\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.761612 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761511    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8143ec7-9638-4e96-87eb-efe88a8ca5ec-host-slash\") pod \"iptables-alerter-9hzfd\" (UID: \"b8143ec7-9638-4e96-87eb-efe88a8ca5ec\") " pod="openshift-network-operator/iptables-alerter-9hzfd"
Jul 30 02:26:05.761612 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761535    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-system-cni-dir\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.761612 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761559    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-multus-socket-dir-parent\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.761612 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761585    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b34ac4fa-e29d-4585-80fe-928a134067aa-cni-binary-copy\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.761612 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761611    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-sysconfig\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.761935 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761657    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-lib-modules\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.761935 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761688    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-ovn-node-metrics-cert\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.761935 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761721    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/228a0670-c32b-4bf3-a593-4e900bae075b-etc-selinux\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.761935 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761749    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/228a0670-c32b-4bf3-a593-4e900bae075b-sys-fs\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.761935 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761775    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-cnibin\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.761935 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761798    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-host-run-multus-certs\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.761935 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761822    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b34ac4fa-e29d-4585-80fe-928a134067aa-os-release\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.761935 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761848    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b34ac4fa-e29d-4585-80fe-928a134067aa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.761935 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761886    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-run\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.761935 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761916    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6mzq\" (UniqueName: \"kubernetes.io/projected/e5d1eddb-bec6-4468-9218-e850a9214f81-kube-api-access-b6mzq\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.762384 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761941    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-modprobe-d\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.762384 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.761975    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b34ac4fa-e29d-4585-80fe-928a134067aa-system-cni-dir\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.762384 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762022    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b34ac4fa-e29d-4585-80fe-928a134067aa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.762384 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762069    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs\") pod \"network-metrics-daemon-wphj8\" (UID: \"ec397db7-4d16-4a1d-9ace-fb4de6e17039\") " pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:05.762384 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762091    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-run-netns\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.762384 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762114    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-run-openvswitch\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.762384 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762137    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a7211fce-5509-4ca2-8fb1-33a73122baec-konnectivity-ca\") pod \"konnectivity-agent-b5trq\" (UID: \"a7211fce-5509-4ca2-8fb1-33a73122baec\") " pod="kube-system/konnectivity-agent-b5trq"
Jul 30 02:26:05.762384 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762159    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/648853ed-378d-4c9d-82e8-5cbb958399db-tmp-dir\") pod \"node-resolver-bzccb\" (UID: \"648853ed-378d-4c9d-82e8-5cbb958399db\") " pod="openshift-dns/node-resolver-bzccb"
Jul 30 02:26:05.762384 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762201    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc2726a6-0c55-49f0-823e-58839b890c37-multus-daemon-config\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.762384 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762224    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b34ac4fa-e29d-4585-80fe-928a134067aa-cnibin\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.762384 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762246    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a7211fce-5509-4ca2-8fb1-33a73122baec-agent-certs\") pod \"konnectivity-agent-b5trq\" (UID: \"a7211fce-5509-4ca2-8fb1-33a73122baec\") " pod="kube-system/konnectivity-agent-b5trq"
Jul 30 02:26:05.762384 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762279    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmtdp\" (UniqueName: \"kubernetes.io/projected/228a0670-c32b-4bf3-a593-4e900bae075b-kube-api-access-gmtdp\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.762384 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762301    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-cni-netd\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.762384 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762351    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret\") pod \"global-pull-secret-syncer-fsv7s\" (UID: \"30a8139c-c65b-45ea-aa17-645a0e4c634a\") " pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:05.762384 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762387    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-host-run-k8s-cni-cncf-io\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.762959 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:05.762459    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 02:26:05.762959 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762481    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6swb\" (UniqueName: \"kubernetes.io/projected/ec397db7-4d16-4a1d-9ace-fb4de6e17039-kube-api-access-j6swb\") pod \"network-metrics-daemon-wphj8\" (UID: \"ec397db7-4d16-4a1d-9ace-fb4de6e17039\") " pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:05.762959 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:05.762527    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret podName:30a8139c-c65b-45ea-aa17-645a0e4c634a nodeName:}" failed. No retries permitted until 2026-07-30 02:26:06.262510423 +0000 UTC m=+1.909293806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret") pod "global-pull-secret-syncer-fsv7s" (UID: "30a8139c-c65b-45ea-aa17-645a0e4c634a") : object "kube-system"/"original-pull-secret" not registered
Jul 30 02:26:05.762959 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762587    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-systemd\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.762959 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762614    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-etc-openvswitch\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.762959 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762638    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpwkn\" (UniqueName: \"kubernetes.io/projected/648853ed-378d-4c9d-82e8-5cbb958399db-kube-api-access-bpwkn\") pod \"node-resolver-bzccb\" (UID: \"648853ed-378d-4c9d-82e8-5cbb958399db\") " pod="openshift-dns/node-resolver-bzccb"
Jul 30 02:26:05.762959 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762666    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcd6r\" (UniqueName: \"kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r\") pod \"network-check-target-w6w7c\" (UID: \"5edd31cd-4288-4f9f-a1a0-a840455c53f2\") " pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:05.762959 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762708    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/228a0670-c32b-4bf3-a593-4e900bae075b-device-dir\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.762959 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762734    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-cni-bin\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.762959 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762757    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-host-var-lib-kubelet\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.762959 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762808    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-etc-kubernetes\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.762959 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4lbb\" (UniqueName: \"kubernetes.io/projected/b34ac4fa-e29d-4585-80fe-928a134067aa-kube-api-access-k4lbb\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.762959 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762878    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-tuned\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.762959 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762925    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/30a8139c-c65b-45ea-aa17-645a0e4c634a-dbus\") pod \"global-pull-secret-syncer-fsv7s\" (UID: \"30a8139c-c65b-45ea-aa17-645a0e4c634a\") " pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:05.763499 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.762964    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54746\" (UniqueName: \"kubernetes.io/projected/b8143ec7-9638-4e96-87eb-efe88a8ca5ec-kube-api-access-54746\") pod \"iptables-alerter-9hzfd\" (UID: \"b8143ec7-9638-4e96-87eb-efe88a8ca5ec\") " pod="openshift-network-operator/iptables-alerter-9hzfd"
Jul 30 02:26:05.763499 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763004    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-host-var-lib-cni-multus\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.763499 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763060    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b34ac4fa-e29d-4585-80fe-928a134067aa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.763499 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763088    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-sysctl-d\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.763499 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763106    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/228a0670-c32b-4bf3-a593-4e900bae075b-socket-dir\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.763499 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763136    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-multus-cni-dir\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.763499 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763187    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/30a8139c-c65b-45ea-aa17-645a0e4c634a-dbus\") pod \"global-pull-secret-syncer-fsv7s\" (UID: \"30a8139c-c65b-45ea-aa17-645a0e4c634a\") " pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:05.763499 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763200    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-host-run-netns\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.763499 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-multus-conf-dir\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.763499 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763255    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsb7q\" (UniqueName: \"kubernetes.io/projected/fc2726a6-0c55-49f0-823e-58839b890c37-kube-api-access-qsb7q\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.763499 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763277    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-systemd-units\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.763499 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763302    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.763499 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n8kw\" (UniqueName: \"kubernetes.io/projected/1f1a6683-1bcf-48ce-a20b-525984e9a23a-kube-api-access-7n8kw\") pod \"node-ca-bspvc\" (UID: \"1f1a6683-1bcf-48ce-a20b-525984e9a23a\") " pod="openshift-image-registry/node-ca-bspvc"
Jul 30 02:26:05.763499 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763348    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-kubelet\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.763499 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763361    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-run-systemd\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.763499 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763376    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/648853ed-378d-4c9d-82e8-5cbb958399db-hosts-file\") pod \"node-resolver-bzccb\" (UID: \"648853ed-378d-4c9d-82e8-5cbb958399db\") " pod="openshift-dns/node-resolver-bzccb"
Jul 30 02:26:05.764061 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763401    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f1a6683-1bcf-48ce-a20b-525984e9a23a-serviceca\") pod \"node-ca-bspvc\" (UID: \"1f1a6683-1bcf-48ce-a20b-525984e9a23a\") " pod="openshift-image-registry/node-ca-bspvc"
Jul 30 02:26:05.764061 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763421    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/30a8139c-c65b-45ea-aa17-645a0e4c634a-kubelet-config\") pod \"global-pull-secret-syncer-fsv7s\" (UID: \"30a8139c-c65b-45ea-aa17-645a0e4c634a\") " pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:05.764061 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763438    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8143ec7-9638-4e96-87eb-efe88a8ca5ec-iptables-alerter-script\") pod \"iptables-alerter-9hzfd\" (UID: \"b8143ec7-9638-4e96-87eb-efe88a8ca5ec\") " pod="openshift-network-operator/iptables-alerter-9hzfd"
Jul 30 02:26:05.764061 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763478    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-host-var-lib-cni-bin\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.764061 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763483    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/30a8139c-c65b-45ea-aa17-645a0e4c634a-kubelet-config\") pod \"global-pull-secret-syncer-fsv7s\" (UID: \"30a8139c-c65b-45ea-aa17-645a0e4c634a\") " pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:05.764061 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-hostroot\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.764061 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763557    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-run-ovn\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.764061 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763618    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-run-ovn-kubernetes\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.764061 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763637    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-env-overrides\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.764061 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763661    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f1a6683-1bcf-48ce-a20b-525984e9a23a-host\") pod \"node-ca-bspvc\" (UID: \"1f1a6683-1bcf-48ce-a20b-525984e9a23a\") " pod="openshift-image-registry/node-ca-bspvc"
Jul 30 02:26:05.764061 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763686    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc2726a6-0c55-49f0-823e-58839b890c37-cni-binary-copy\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.764061 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763723    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-sysctl-conf\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.764061 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763748    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-var-lib-kubelet\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.764061 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763775    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-host\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.764061 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.763790    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5d1eddb-bec6-4468-9218-e850a9214f81-tmp\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.764585 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.764295    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-var-lib-openvswitch\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.764585 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.764328    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-node-log\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.764585 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.764366    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-ovnkube-config\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.764585 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.764393    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-log-socket\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.764585 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.764416    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-ovnkube-script-lib\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.764585 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.764448    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxclz\" (UniqueName: \"kubernetes.io/projected/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-kube-api-access-hxclz\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.770258 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:05.770233    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 02:26:05.770258 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:05.770256    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 02:26:05.770258 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:05.770266    2565 projected.go:196] Error preparing data for projected volume kube-api-access-dcd6r for pod openshift-network-diagnostics/network-check-target-w6w7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 02:26:05.770494 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:05.770312    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r podName:5edd31cd-4288-4f9f-a1a0-a840455c53f2 nodeName:}" failed. No retries permitted until 2026-07-30 02:26:06.270299267 +0000 UTC m=+1.917082654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dcd6r" (UniqueName: "kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r") pod "network-check-target-w6w7c" (UID: "5edd31cd-4288-4f9f-a1a0-a840455c53f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 02:26:05.862940 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:05.862913    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d948a8b8f6069f9ae71aaf0197e360.slice/crio-f51cbd5c82d3975a20f5047645922f055ce94ec23bdd717c3140b73df1ee1042 WatchSource:0}: Error finding container f51cbd5c82d3975a20f5047645922f055ce94ec23bdd717c3140b73df1ee1042: Status 404 returned error can't find the container with id f51cbd5c82d3975a20f5047645922f055ce94ec23bdd717c3140b73df1ee1042
Jul 30 02:26:05.864851 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.864829    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/228a0670-c32b-4bf3-a593-4e900bae075b-socket-dir\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.864910 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.864868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-multus-cni-dir\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.864910 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.864892    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-host-run-netns\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.864910 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.864908    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-multus-conf-dir\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.865013 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.864923    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsb7q\" (UniqueName: \"kubernetes.io/projected/fc2726a6-0c55-49f0-823e-58839b890c37-kube-api-access-qsb7q\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.865013 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.864941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-systemd-units\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.865013 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.864966    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.865013 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.864968    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-multus-cni-dir\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.865013 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.864983    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-multus-conf-dir\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.865013 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.864986    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/228a0670-c32b-4bf3-a593-4e900bae075b-socket-dir\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.865013 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.864993    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n8kw\" (UniqueName: \"kubernetes.io/projected/1f1a6683-1bcf-48ce-a20b-525984e9a23a-kube-api-access-7n8kw\") pod \"node-ca-bspvc\" (UID: \"1f1a6683-1bcf-48ce-a20b-525984e9a23a\") " pod="openshift-image-registry/node-ca-bspvc"
Jul 30 02:26:05.865286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865031    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-host-run-netns\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.865286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865040    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-kubelet\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.865286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865058    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-systemd-units\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.865286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865065    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-run-systemd\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.865286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865071    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.865286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865098    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-run-systemd\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.865286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865099    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/648853ed-378d-4c9d-82e8-5cbb958399db-hosts-file\") pod \"node-resolver-bzccb\" (UID: \"648853ed-378d-4c9d-82e8-5cbb958399db\") " pod="openshift-dns/node-resolver-bzccb"
Jul 30 02:26:05.865286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865110    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-kubelet\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.865286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865130    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f1a6683-1bcf-48ce-a20b-525984e9a23a-serviceca\") pod \"node-ca-bspvc\" (UID: \"1f1a6683-1bcf-48ce-a20b-525984e9a23a\") " pod="openshift-image-registry/node-ca-bspvc"
Jul 30 02:26:05.865286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865139    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/648853ed-378d-4c9d-82e8-5cbb958399db-hosts-file\") pod \"node-resolver-bzccb\" (UID: \"648853ed-378d-4c9d-82e8-5cbb958399db\") " pod="openshift-dns/node-resolver-bzccb"
Jul 30 02:26:05.865286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865156    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8143ec7-9638-4e96-87eb-efe88a8ca5ec-iptables-alerter-script\") pod \"iptables-alerter-9hzfd\" (UID: \"b8143ec7-9638-4e96-87eb-efe88a8ca5ec\") " pod="openshift-network-operator/iptables-alerter-9hzfd"
Jul 30 02:26:05.865286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865205    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-host-var-lib-cni-bin\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.865286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865231    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-hostroot\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.865286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865254    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-run-ovn\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.865286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865260    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-host-var-lib-cni-bin\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.865286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865279    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-run-ovn-kubernetes\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.865767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865305    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-hostroot\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.865767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865311    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-run-ovn\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.865767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865321    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-env-overrides\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.865767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865346    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f1a6683-1bcf-48ce-a20b-525984e9a23a-host\") pod \"node-ca-bspvc\" (UID: \"1f1a6683-1bcf-48ce-a20b-525984e9a23a\") " pod="openshift-image-registry/node-ca-bspvc"
Jul 30 02:26:05.865767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865350    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-run-ovn-kubernetes\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.865767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865370    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc2726a6-0c55-49f0-823e-58839b890c37-cni-binary-copy\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.865767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865390    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f1a6683-1bcf-48ce-a20b-525984e9a23a-host\") pod \"node-ca-bspvc\" (UID: \"1f1a6683-1bcf-48ce-a20b-525984e9a23a\") " pod="openshift-image-registry/node-ca-bspvc"
Jul 30 02:26:05.865767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865393    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-sysctl-conf\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.865767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865425    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-var-lib-kubelet\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.865767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-host\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.865767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5d1eddb-bec6-4468-9218-e850a9214f81-tmp\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.865767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865493    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-sysctl-conf\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.865767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865494    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-var-lib-openvswitch\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.865767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865519    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-node-log\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.865767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865543    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-ovnkube-config\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.865767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865567    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-log-socket\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.865767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865591    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-ovnkube-script-lib\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.865767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865650    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxclz\" (UniqueName: \"kubernetes.io/projected/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-kube-api-access-hxclz\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.866328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865665    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f1a6683-1bcf-48ce-a20b-525984e9a23a-serviceca\") pod \"node-ca-bspvc\" (UID: \"1f1a6683-1bcf-48ce-a20b-525984e9a23a\") " pod="openshift-image-registry/node-ca-bspvc"
Jul 30 02:26:05.866328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865681    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/228a0670-c32b-4bf3-a593-4e900bae075b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.866328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865705    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/228a0670-c32b-4bf3-a593-4e900bae075b-registration-dir\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.866328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865728    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-os-release\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.866328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865760    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8143ec7-9638-4e96-87eb-efe88a8ca5ec-iptables-alerter-script\") pod \"iptables-alerter-9hzfd\" (UID: \"b8143ec7-9638-4e96-87eb-efe88a8ca5ec\") " pod="openshift-network-operator/iptables-alerter-9hzfd"
Jul 30 02:26:05.866328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865789    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-env-overrides\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.866328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865813    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-var-lib-kubelet\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.866328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865766    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-kubernetes\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.866328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865823    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-kubernetes\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.866328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865846    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-sys\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.866328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865860    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-host\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.866328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865870    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-slash\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.866328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865893    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-log-socket\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.866328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865893    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8143ec7-9638-4e96-87eb-efe88a8ca5ec-host-slash\") pod \"iptables-alerter-9hzfd\" (UID: \"b8143ec7-9638-4e96-87eb-efe88a8ca5ec\") " pod="openshift-network-operator/iptables-alerter-9hzfd"
Jul 30 02:26:05.866328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865891    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc2726a6-0c55-49f0-823e-58839b890c37-cni-binary-copy\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.866328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865928    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8143ec7-9638-4e96-87eb-efe88a8ca5ec-host-slash\") pod \"iptables-alerter-9hzfd\" (UID: \"b8143ec7-9638-4e96-87eb-efe88a8ca5ec\") " pod="openshift-network-operator/iptables-alerter-9hzfd"
Jul 30 02:26:05.866328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865969    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-system-cni-dir\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.866328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865968    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-var-lib-openvswitch\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.867056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865976    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-node-log\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.867056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866005    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-sys\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.867056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866042    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/228a0670-c32b-4bf3-a593-4e900bae075b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.867056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866050    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-slash\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.867056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.865929    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-system-cni-dir\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.867056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866088    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-multus-socket-dir-parent\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.867056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866109    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/228a0670-c32b-4bf3-a593-4e900bae075b-registration-dir\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.867056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866112    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b34ac4fa-e29d-4585-80fe-928a134067aa-cni-binary-copy\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.867056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866141    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-sysconfig\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.867056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866205    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-lib-modules\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.867056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866235    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-ovn-node-metrics-cert\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.867056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866258    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/228a0670-c32b-4bf3-a593-4e900bae075b-etc-selinux\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.867056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866278    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/228a0670-c32b-4bf3-a593-4e900bae075b-sys-fs\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.867056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866299    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-cnibin\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.867056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866340    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-host-run-multus-certs\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.867056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866358    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-ovnkube-script-lib\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.867056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b34ac4fa-e29d-4585-80fe-928a134067aa-os-release\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.867658 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866390    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b34ac4fa-e29d-4585-80fe-928a134067aa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.867658 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866416    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-run\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.867658 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866423    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-multus-socket-dir-parent\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.867658 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866433    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-os-release\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.867658 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6mzq\" (UniqueName: \"kubernetes.io/projected/e5d1eddb-bec6-4468-9218-e850a9214f81-kube-api-access-b6mzq\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.867658 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866465    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-modprobe-d\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.867658 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866470    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/228a0670-c32b-4bf3-a593-4e900bae075b-sys-fs\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.867658 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866489    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b34ac4fa-e29d-4585-80fe-928a134067aa-system-cni-dir\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.867658 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866497    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-ovnkube-config\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.867658 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866512    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b34ac4fa-e29d-4585-80fe-928a134067aa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.867658 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866520    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-lib-modules\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.867658 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866536    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs\") pod \"network-metrics-daemon-wphj8\" (UID: \"ec397db7-4d16-4a1d-9ace-fb4de6e17039\") " pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:05.867658 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866561    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-run-netns\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.867658 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866575    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/228a0670-c32b-4bf3-a593-4e900bae075b-etc-selinux\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.867658 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866584    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-run-openvswitch\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.867658 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866608    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a7211fce-5509-4ca2-8fb1-33a73122baec-konnectivity-ca\") pod \"konnectivity-agent-b5trq\" (UID: \"a7211fce-5509-4ca2-8fb1-33a73122baec\") " pod="kube-system/konnectivity-agent-b5trq"
Jul 30 02:26:05.867658 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866632    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/648853ed-378d-4c9d-82e8-5cbb958399db-tmp-dir\") pod \"node-resolver-bzccb\" (UID: \"648853ed-378d-4c9d-82e8-5cbb958399db\") " pod="openshift-dns/node-resolver-bzccb"
Jul 30 02:26:05.868625 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866643    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-cnibin\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.868625 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866655    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc2726a6-0c55-49f0-823e-58839b890c37-multus-daemon-config\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.868625 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866667    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-host-run-multus-certs\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.868625 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866680    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b34ac4fa-e29d-4585-80fe-928a134067aa-cni-binary-copy\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.868625 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866679    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b34ac4fa-e29d-4585-80fe-928a134067aa-cnibin\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.868625 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866703    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b34ac4fa-e29d-4585-80fe-928a134067aa-os-release\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.868625 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866710    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a7211fce-5509-4ca2-8fb1-33a73122baec-agent-certs\") pod \"konnectivity-agent-b5trq\" (UID: \"a7211fce-5509-4ca2-8fb1-33a73122baec\") " pod="kube-system/konnectivity-agent-b5trq"
Jul 30 02:26:05.868625 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866731    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-run\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.868625 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866746    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b34ac4fa-e29d-4585-80fe-928a134067aa-system-cni-dir\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.868625 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866743    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmtdp\" (UniqueName: \"kubernetes.io/projected/228a0670-c32b-4bf3-a593-4e900bae075b-kube-api-access-gmtdp\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.868625 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866791    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-cni-netd\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.868625 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866818    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-run-openvswitch\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.868625 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866829    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-host-run-k8s-cni-cncf-io\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.868625 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866852    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6swb\" (UniqueName: \"kubernetes.io/projected/ec397db7-4d16-4a1d-9ace-fb4de6e17039-kube-api-access-j6swb\") pod \"network-metrics-daemon-wphj8\" (UID: \"ec397db7-4d16-4a1d-9ace-fb4de6e17039\") " pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:05.868625 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866871    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b34ac4fa-e29d-4585-80fe-928a134067aa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.868625 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866873    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-systemd\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.868625 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866913    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-systemd\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.869442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866916    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-etc-openvswitch\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.869442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866744    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 30 02:26:05.869442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866960    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpwkn\" (UniqueName: \"kubernetes.io/projected/648853ed-378d-4c9d-82e8-5cbb958399db-kube-api-access-bpwkn\") pod \"node-resolver-bzccb\" (UID: \"648853ed-378d-4c9d-82e8-5cbb958399db\") " pod="openshift-dns/node-resolver-bzccb"
Jul 30 02:26:05.869442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867000    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/228a0670-c32b-4bf3-a593-4e900bae075b-device-dir\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.869442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867035    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-cni-bin\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.869442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867062    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-host-var-lib-kubelet\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.869442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867086    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-etc-kubernetes\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.869442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867110    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4lbb\" (UniqueName: \"kubernetes.io/projected/b34ac4fa-e29d-4585-80fe-928a134067aa-kube-api-access-k4lbb\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.869442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867135    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-tuned\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.869442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867144    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b34ac4fa-e29d-4585-80fe-928a134067aa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.869442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866611    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-sysconfig\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.869442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.866792    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-modprobe-d\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.869442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867268    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/228a0670-c32b-4bf3-a593-4e900bae075b-device-dir\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.869442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867279    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a7211fce-5509-4ca2-8fb1-33a73122baec-konnectivity-ca\") pod \"konnectivity-agent-b5trq\" (UID: \"a7211fce-5509-4ca2-8fb1-33a73122baec\") " pod="kube-system/konnectivity-agent-b5trq"
Jul 30 02:26:05.869442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867349    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 02:26:05.869442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867362    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc2726a6-0c55-49f0-823e-58839b890c37-multus-daemon-config\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.869442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867382    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-run-netns\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.869442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867419    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-etc-openvswitch\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.870293 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867455    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b34ac4fa-e29d-4585-80fe-928a134067aa-cnibin\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.870293 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867504    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-cni-bin\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.870293 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867544    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-host-var-lib-kubelet\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.870293 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867551    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/648853ed-378d-4c9d-82e8-5cbb958399db-tmp-dir\") pod \"node-resolver-bzccb\" (UID: \"648853ed-378d-4c9d-82e8-5cbb958399db\") " pod="openshift-dns/node-resolver-bzccb"
Jul 30 02:26:05.870293 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867632    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-host-cni-netd\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.870293 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867640    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-etc-kubernetes\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.870293 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867893    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-host-run-k8s-cni-cncf-io\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.870293 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:05.867352    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 02:26:05.870293 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:05.868015    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs podName:ec397db7-4d16-4a1d-9ace-fb4de6e17039 nodeName:}" failed. No retries permitted until 2026-07-30 02:26:06.367996701 +0000 UTC m=+2.014780105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs") pod "network-metrics-daemon-wphj8" (UID: "ec397db7-4d16-4a1d-9ace-fb4de6e17039") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 02:26:05.870293 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.867159    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54746\" (UniqueName: \"kubernetes.io/projected/b8143ec7-9638-4e96-87eb-efe88a8ca5ec-kube-api-access-54746\") pod \"iptables-alerter-9hzfd\" (UID: \"b8143ec7-9638-4e96-87eb-efe88a8ca5ec\") " pod="openshift-network-operator/iptables-alerter-9hzfd"
Jul 30 02:26:05.870293 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.868242    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-host-var-lib-cni-multus\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.870293 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.868272    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b34ac4fa-e29d-4585-80fe-928a134067aa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.870293 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.868298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-sysctl-d\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.870293 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.868366    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc2726a6-0c55-49f0-823e-58839b890c37-host-var-lib-cni-multus\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.870293 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.868407    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-sysctl-d\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.870293 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.868738    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b34ac4fa-e29d-4585-80fe-928a134067aa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.870922 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.870493    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e5d1eddb-bec6-4468-9218-e850a9214f81-etc-tuned\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.870922 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.870637    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-ovn-node-metrics-cert\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.870922 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.870763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a7211fce-5509-4ca2-8fb1-33a73122baec-agent-certs\") pod \"konnectivity-agent-b5trq\" (UID: \"a7211fce-5509-4ca2-8fb1-33a73122baec\") " pod="kube-system/konnectivity-agent-b5trq"
Jul 30 02:26:05.870922 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.870790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5d1eddb-bec6-4468-9218-e850a9214f81-tmp\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.877371 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.877347    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n8kw\" (UniqueName: \"kubernetes.io/projected/1f1a6683-1bcf-48ce-a20b-525984e9a23a-kube-api-access-7n8kw\") pod \"node-ca-bspvc\" (UID: \"1f1a6683-1bcf-48ce-a20b-525984e9a23a\") " pod="openshift-image-registry/node-ca-bspvc"
Jul 30 02:26:05.878742 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.878720    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6mzq\" (UniqueName: \"kubernetes.io/projected/e5d1eddb-bec6-4468-9218-e850a9214f81-kube-api-access-b6mzq\") pod \"tuned-gjr85\" (UID: \"e5d1eddb-bec6-4468-9218-e850a9214f81\") " pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:05.878838 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.878791    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54746\" (UniqueName: \"kubernetes.io/projected/b8143ec7-9638-4e96-87eb-efe88a8ca5ec-kube-api-access-54746\") pod \"iptables-alerter-9hzfd\" (UID: \"b8143ec7-9638-4e96-87eb-efe88a8ca5ec\") " pod="openshift-network-operator/iptables-alerter-9hzfd"
Jul 30 02:26:05.879336 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.879308    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxclz\" (UniqueName: \"kubernetes.io/projected/3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b-kube-api-access-hxclz\") pod \"ovnkube-node-6rrn8\" (UID: \"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:05.879473 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.879452    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpwkn\" (UniqueName: \"kubernetes.io/projected/648853ed-378d-4c9d-82e8-5cbb958399db-kube-api-access-bpwkn\") pod \"node-resolver-bzccb\" (UID: \"648853ed-378d-4c9d-82e8-5cbb958399db\") " pod="openshift-dns/node-resolver-bzccb"
Jul 30 02:26:05.879857 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.879837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4lbb\" (UniqueName: \"kubernetes.io/projected/b34ac4fa-e29d-4585-80fe-928a134067aa-kube-api-access-k4lbb\") pod \"multus-additional-cni-plugins-278lc\" (UID: \"b34ac4fa-e29d-4585-80fe-928a134067aa\") " pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:05.879932 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.879877    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsb7q\" (UniqueName: \"kubernetes.io/projected/fc2726a6-0c55-49f0-823e-58839b890c37-kube-api-access-qsb7q\") pod \"multus-77n5d\" (UID: \"fc2726a6-0c55-49f0-823e-58839b890c37\") " pod="openshift-multus/multus-77n5d"
Jul 30 02:26:05.879932 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.879918    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6swb\" (UniqueName: \"kubernetes.io/projected/ec397db7-4d16-4a1d-9ace-fb4de6e17039-kube-api-access-j6swb\") pod \"network-metrics-daemon-wphj8\" (UID: \"ec397db7-4d16-4a1d-9ace-fb4de6e17039\") " pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:05.880381 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.880366    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmtdp\" (UniqueName: \"kubernetes.io/projected/228a0670-c32b-4bf3-a593-4e900bae075b-kube-api-access-gmtdp\") pod \"aws-ebs-csi-driver-node-phsz6\" (UID: \"228a0670-c32b-4bf3-a593-4e900bae075b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:05.993927 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:05.993842    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6"
Jul 30 02:26:06.005636 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:06.005611    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228a0670_c32b_4bf3_a593_4e900bae075b.slice/crio-f34421d4f7f3b701246fe10c9da6fadb1ec4b34e64a6b3d8bc73aec28b033089 WatchSource:0}: Error finding container f34421d4f7f3b701246fe10c9da6fadb1ec4b34e64a6b3d8bc73aec28b033089: Status 404 returned error can't find the container with id f34421d4f7f3b701246fe10c9da6fadb1ec4b34e64a6b3d8bc73aec28b033089
Jul 30 02:26:06.008890 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.008865    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 02:26:06.014533 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.014516    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gjr85"
Jul 30 02:26:06.022489 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:06.022465    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d1eddb_bec6_4468_9218_e850a9214f81.slice/crio-5b20badad753d2dfe177004e461f17eb61317f5c11d8ff0b6624f3ef709c46e4 WatchSource:0}: Error finding container 5b20badad753d2dfe177004e461f17eb61317f5c11d8ff0b6624f3ef709c46e4: Status 404 returned error can't find the container with id 5b20badad753d2dfe177004e461f17eb61317f5c11d8ff0b6624f3ef709c46e4
Jul 30 02:26:06.036178 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.036140    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9hzfd"
Jul 30 02:26:06.043886 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:06.043863    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8143ec7_9638_4e96_87eb_efe88a8ca5ec.slice/crio-6dc5d1459627d317b7c04125fd149138b882e4aa7a1feb5ccc71c1329ee7b57d WatchSource:0}: Error finding container 6dc5d1459627d317b7c04125fd149138b882e4aa7a1feb5ccc71c1329ee7b57d: Status 404 returned error can't find the container with id 6dc5d1459627d317b7c04125fd149138b882e4aa7a1feb5ccc71c1329ee7b57d
Jul 30 02:26:06.046860 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.046840    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:06.054497 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:06.054475    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5bcf49_0c5c_4a7b_83ba_7bc1167f730b.slice/crio-007e08bb257db9b96d98ffd429e40f1025bea9661ca72a12c71a895c067a6ff7 WatchSource:0}: Error finding container 007e08bb257db9b96d98ffd429e40f1025bea9661ca72a12c71a895c067a6ff7: Status 404 returned error can't find the container with id 007e08bb257db9b96d98ffd429e40f1025bea9661ca72a12c71a895c067a6ff7
Jul 30 02:26:06.060930 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.060915    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b5trq"
Jul 30 02:26:06.069557 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:06.069538    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7211fce_5509_4ca2_8fb1_33a73122baec.slice/crio-d028c441af31065054d6dd1187fdc0f3c6752c8c547373f2c8b49f6a2c135eff WatchSource:0}: Error finding container d028c441af31065054d6dd1187fdc0f3c6752c8c547373f2c8b49f6a2c135eff: Status 404 returned error can't find the container with id d028c441af31065054d6dd1187fdc0f3c6752c8c547373f2c8b49f6a2c135eff
Jul 30 02:26:06.073046 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.073030    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bzccb"
Jul 30 02:26:06.073131 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.073117    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 02:26:06.082187 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.082149    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bspvc"
Jul 30 02:26:06.082906 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:06.082887    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648853ed_378d_4c9d_82e8_5cbb958399db.slice/crio-b2294950e05f27781e2bfe0d2c97121a5749ce3e1e2350a5d0932acf4524a2f5 WatchSource:0}: Error finding container b2294950e05f27781e2bfe0d2c97121a5749ce3e1e2350a5d0932acf4524a2f5: Status 404 returned error can't find the container with id b2294950e05f27781e2bfe0d2c97121a5749ce3e1e2350a5d0932acf4524a2f5
Jul 30 02:26:06.090947 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:06.090922    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1a6683_1bcf_48ce_a20b_525984e9a23a.slice/crio-9780034e1d4ab29b67f240a49bc87f1b3cff150540b2b1892faa0d73a6e6afef WatchSource:0}: Error finding container 9780034e1d4ab29b67f240a49bc87f1b3cff150540b2b1892faa0d73a6e6afef: Status 404 returned error can't find the container with id 9780034e1d4ab29b67f240a49bc87f1b3cff150540b2b1892faa0d73a6e6afef
Jul 30 02:26:06.107605 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.107586    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-77n5d"
Jul 30 02:26:06.111284 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.111263    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-278lc"
Jul 30 02:26:06.117770 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:06.117740    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2726a6_0c55_49f0_823e_58839b890c37.slice/crio-aa70f855b2f91455b233147060e1fec651452c2469c8945dd01be2aca1253e2a WatchSource:0}: Error finding container aa70f855b2f91455b233147060e1fec651452c2469c8945dd01be2aca1253e2a: Status 404 returned error can't find the container with id aa70f855b2f91455b233147060e1fec651452c2469c8945dd01be2aca1253e2a
Jul 30 02:26:06.121063 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:06.121042    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34ac4fa_e29d_4585_80fe_928a134067aa.slice/crio-3a687185c46714d625b48e430a78871e86d302987ed6bce10849278d00e5e33b WatchSource:0}: Error finding container 3a687185c46714d625b48e430a78871e86d302987ed6bce10849278d00e5e33b: Status 404 returned error can't find the container with id 3a687185c46714d625b48e430a78871e86d302987ed6bce10849278d00e5e33b
Jul 30 02:26:06.270648 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.270626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret\") pod \"global-pull-secret-syncer-fsv7s\" (UID: \"30a8139c-c65b-45ea-aa17-645a0e4c634a\") " pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:06.270790 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.270659    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcd6r\" (UniqueName: \"kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r\") pod \"network-check-target-w6w7c\" (UID: \"5edd31cd-4288-4f9f-a1a0-a840455c53f2\") " pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:06.270829 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:06.270809    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 02:26:06.270829 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:06.270813    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 02:26:06.270829 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:06.270823    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 02:26:06.270921 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:06.270837    2565 projected.go:196] Error preparing data for projected volume kube-api-access-dcd6r for pod openshift-network-diagnostics/network-check-target-w6w7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 02:26:06.270921 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:06.270882    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r podName:5edd31cd-4288-4f9f-a1a0-a840455c53f2 nodeName:}" failed. No retries permitted until 2026-07-30 02:26:07.270868695 +0000 UTC m=+2.917652079 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dcd6r" (UniqueName: "kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r") pod "network-check-target-w6w7c" (UID: "5edd31cd-4288-4f9f-a1a0-a840455c53f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 02:26:06.270921 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:06.270897    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret podName:30a8139c-c65b-45ea-aa17-645a0e4c634a nodeName:}" failed. No retries permitted until 2026-07-30 02:26:07.270890133 +0000 UTC m=+2.917673518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret") pod "global-pull-secret-syncer-fsv7s" (UID: "30a8139c-c65b-45ea-aa17-645a0e4c634a") : object "kube-system"/"original-pull-secret" not registered
Jul 30 02:26:06.271600 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:06.271574    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2615c8560d8f795d489b67609c7125ee.slice/crio-d3b349db845561a94ea3f74467c8ecac3cf44b3eb1c4f4acbe68a645547cd11b WatchSource:0}: Error finding container d3b349db845561a94ea3f74467c8ecac3cf44b3eb1c4f4acbe68a645547cd11b: Status 404 returned error can't find the container with id d3b349db845561a94ea3f74467c8ecac3cf44b3eb1c4f4acbe68a645547cd11b
Jul 30 02:26:06.371928 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.371899    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs\") pod \"network-metrics-daemon-wphj8\" (UID: \"ec397db7-4d16-4a1d-9ace-fb4de6e17039\") " pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:06.372094 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:06.372029    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 02:26:06.372094 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:06.372081    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs podName:ec397db7-4d16-4a1d-9ace-fb4de6e17039 nodeName:}" failed. No retries permitted until 2026-07-30 02:26:07.372068487 +0000 UTC m=+3.018851872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs") pod "network-metrics-daemon-wphj8" (UID: "ec397db7-4d16-4a1d-9ace-fb4de6e17039") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 02:26:06.707877 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.707758    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 02:21:05 +0000 UTC" deadline="2028-03-29 15:16:18.788082149 +0000 UTC"
Jul 30 02:26:06.707877 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.707825    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14604h50m12.080261456s"
Jul 30 02:26:06.796168 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.796136    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:06.796374 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:06.796302    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:26:06.823743 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.823639    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-89.ec2.internal" event={"ID":"2615c8560d8f795d489b67609c7125ee","Type":"ContainerStarted","Data":"d3b349db845561a94ea3f74467c8ecac3cf44b3eb1c4f4acbe68a645547cd11b"}
Jul 30 02:26:06.826683 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.825718    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-77n5d" event={"ID":"fc2726a6-0c55-49f0-823e-58839b890c37","Type":"ContainerStarted","Data":"aa70f855b2f91455b233147060e1fec651452c2469c8945dd01be2aca1253e2a"}
Jul 30 02:26:06.830464 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.830419    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bzccb" event={"ID":"648853ed-378d-4c9d-82e8-5cbb958399db","Type":"ContainerStarted","Data":"b2294950e05f27781e2bfe0d2c97121a5749ce3e1e2350a5d0932acf4524a2f5"}
Jul 30 02:26:06.834665 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.834637    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b5trq" event={"ID":"a7211fce-5509-4ca2-8fb1-33a73122baec","Type":"ContainerStarted","Data":"d028c441af31065054d6dd1187fdc0f3c6752c8c547373f2c8b49f6a2c135eff"}
Jul 30 02:26:06.836835 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.836811    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8" event={"ID":"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b","Type":"ContainerStarted","Data":"007e08bb257db9b96d98ffd429e40f1025bea9661ca72a12c71a895c067a6ff7"}
Jul 30 02:26:06.841827 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.841803    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9hzfd" event={"ID":"b8143ec7-9638-4e96-87eb-efe88a8ca5ec","Type":"ContainerStarted","Data":"6dc5d1459627d317b7c04125fd149138b882e4aa7a1feb5ccc71c1329ee7b57d"}
Jul 30 02:26:06.846657 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.846631    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gjr85" event={"ID":"e5d1eddb-bec6-4468-9218-e850a9214f81","Type":"ContainerStarted","Data":"5b20badad753d2dfe177004e461f17eb61317f5c11d8ff0b6624f3ef709c46e4"}
Jul 30 02:26:06.854031 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.853724    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6" event={"ID":"228a0670-c32b-4bf3-a593-4e900bae075b","Type":"ContainerStarted","Data":"f34421d4f7f3b701246fe10c9da6fadb1ec4b34e64a6b3d8bc73aec28b033089"}
Jul 30 02:26:06.855910 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.855882    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-278lc" event={"ID":"b34ac4fa-e29d-4585-80fe-928a134067aa","Type":"ContainerStarted","Data":"3a687185c46714d625b48e430a78871e86d302987ed6bce10849278d00e5e33b"}
Jul 30 02:26:06.857751 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.857720    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bspvc" event={"ID":"1f1a6683-1bcf-48ce-a20b-525984e9a23a","Type":"ContainerStarted","Data":"9780034e1d4ab29b67f240a49bc87f1b3cff150540b2b1892faa0d73a6e6afef"}
Jul 30 02:26:06.859342 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.859304    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal" event={"ID":"74d948a8b8f6069f9ae71aaf0197e360","Type":"ContainerStarted","Data":"f51cbd5c82d3975a20f5047645922f055ce94ec23bdd717c3140b73df1ee1042"}
Jul 30 02:26:06.880278 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:06.877135    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 02:26:07.283023 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:07.282944    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret\") pod \"global-pull-secret-syncer-fsv7s\" (UID: \"30a8139c-c65b-45ea-aa17-645a0e4c634a\") " pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:07.283023 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:07.283002    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcd6r\" (UniqueName: \"kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r\") pod \"network-check-target-w6w7c\" (UID: \"5edd31cd-4288-4f9f-a1a0-a840455c53f2\") " pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:07.283320 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:07.283160    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 02:26:07.283320 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:07.283181    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 02:26:07.283320 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:07.283195    2565 projected.go:196] Error preparing data for projected volume kube-api-access-dcd6r for pod openshift-network-diagnostics/network-check-target-w6w7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 02:26:07.283320 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:07.283278    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r podName:5edd31cd-4288-4f9f-a1a0-a840455c53f2 nodeName:}" failed. No retries permitted until 2026-07-30 02:26:09.283253519 +0000 UTC m=+4.930036921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dcd6r" (UniqueName: "kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r") pod "network-check-target-w6w7c" (UID: "5edd31cd-4288-4f9f-a1a0-a840455c53f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 02:26:07.283561 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:07.283361    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 02:26:07.283561 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:07.283402    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret podName:30a8139c-c65b-45ea-aa17-645a0e4c634a nodeName:}" failed. No retries permitted until 2026-07-30 02:26:09.283388495 +0000 UTC m=+4.930171886 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret") pod "global-pull-secret-syncer-fsv7s" (UID: "30a8139c-c65b-45ea-aa17-645a0e4c634a") : object "kube-system"/"original-pull-secret" not registered
Jul 30 02:26:07.383707 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:07.383666    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs\") pod \"network-metrics-daemon-wphj8\" (UID: \"ec397db7-4d16-4a1d-9ace-fb4de6e17039\") " pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:07.383912 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:07.383857    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 02:26:07.383984 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:07.383929    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs podName:ec397db7-4d16-4a1d-9ace-fb4de6e17039 nodeName:}" failed. No retries permitted until 2026-07-30 02:26:09.383908863 +0000 UTC m=+5.030692272 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs") pod "network-metrics-daemon-wphj8" (UID: "ec397db7-4d16-4a1d-9ace-fb4de6e17039") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 02:26:07.708820 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:07.708726    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 02:21:05 +0000 UTC" deadline="2028-02-18 23:35:05.848717981 +0000 UTC"
Jul 30 02:26:07.708820 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:07.708770    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13653h8m58.139952604s"
Jul 30 02:26:07.795310 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:07.795268    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:07.795517 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:07.795426    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsv7s" podUID="30a8139c-c65b-45ea-aa17-645a0e4c634a"
Jul 30 02:26:07.795862 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:07.795836    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:07.795969 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:07.795937    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6w7c" podUID="5edd31cd-4288-4f9f-a1a0-a840455c53f2"
Jul 30 02:26:07.934769 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:07.932778    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 02:26:08.796503 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:08.796468    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:08.796982 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:08.796650    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:26:09.305745 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:09.305705    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret\") pod \"global-pull-secret-syncer-fsv7s\" (UID: \"30a8139c-c65b-45ea-aa17-645a0e4c634a\") " pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:09.305960 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:09.305765    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcd6r\" (UniqueName: \"kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r\") pod \"network-check-target-w6w7c\" (UID: \"5edd31cd-4288-4f9f-a1a0-a840455c53f2\") " pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:09.305960 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:09.305899    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 02:26:09.305960 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:09.305936    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 02:26:09.305960 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:09.305957    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 02:26:09.305960 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:09.305970    2565 projected.go:196] Error preparing data for projected volume kube-api-access-dcd6r for pod openshift-network-diagnostics/network-check-target-w6w7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 02:26:09.306294 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:09.305984    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret podName:30a8139c-c65b-45ea-aa17-645a0e4c634a nodeName:}" failed. No retries permitted until 2026-07-30 02:26:13.305962847 +0000 UTC m=+8.952746242 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret") pod "global-pull-secret-syncer-fsv7s" (UID: "30a8139c-c65b-45ea-aa17-645a0e4c634a") : object "kube-system"/"original-pull-secret" not registered
Jul 30 02:26:09.306294 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:09.306021    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r podName:5edd31cd-4288-4f9f-a1a0-a840455c53f2 nodeName:}" failed. No retries permitted until 2026-07-30 02:26:13.306004356 +0000 UTC m=+8.952787740 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dcd6r" (UniqueName: "kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r") pod "network-check-target-w6w7c" (UID: "5edd31cd-4288-4f9f-a1a0-a840455c53f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 02:26:09.406314 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:09.406278    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs\") pod \"network-metrics-daemon-wphj8\" (UID: \"ec397db7-4d16-4a1d-9ace-fb4de6e17039\") " pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:09.406506 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:09.406475    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 02:26:09.406571 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:09.406536    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs podName:ec397db7-4d16-4a1d-9ace-fb4de6e17039 nodeName:}" failed. No retries permitted until 2026-07-30 02:26:13.406518608 +0000 UTC m=+9.053302008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs") pod "network-metrics-daemon-wphj8" (UID: "ec397db7-4d16-4a1d-9ace-fb4de6e17039") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 02:26:09.794958 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:09.794870    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:09.795145 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:09.795000    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsv7s" podUID="30a8139c-c65b-45ea-aa17-645a0e4c634a"
Jul 30 02:26:09.795336 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:09.795312    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:09.795501 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:09.795420    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6w7c" podUID="5edd31cd-4288-4f9f-a1a0-a840455c53f2"
Jul 30 02:26:10.796351 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:10.796303    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:10.796779 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:10.796455    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:26:11.794881 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:11.794847    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:11.795058 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:11.794970    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsv7s" podUID="30a8139c-c65b-45ea-aa17-645a0e4c634a"
Jul 30 02:26:11.795352 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:11.795333    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:11.795448 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:11.795430    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6w7c" podUID="5edd31cd-4288-4f9f-a1a0-a840455c53f2"
Jul 30 02:26:12.796016 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:12.795981    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:12.796466 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:12.796111    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:26:13.339906 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:13.339867    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret\") pod \"global-pull-secret-syncer-fsv7s\" (UID: \"30a8139c-c65b-45ea-aa17-645a0e4c634a\") " pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:13.340092 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:13.339922    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcd6r\" (UniqueName: \"kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r\") pod \"network-check-target-w6w7c\" (UID: \"5edd31cd-4288-4f9f-a1a0-a840455c53f2\") " pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:13.340092 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:13.340036    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 02:26:13.340092 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:13.340083    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 02:26:13.340286 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:13.340102    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 02:26:13.340286 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:13.340115    2565 projected.go:196] Error preparing data for projected volume kube-api-access-dcd6r for pod openshift-network-diagnostics/network-check-target-w6w7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 02:26:13.340286 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:13.340146    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret podName:30a8139c-c65b-45ea-aa17-645a0e4c634a nodeName:}" failed. No retries permitted until 2026-07-30 02:26:21.340128095 +0000 UTC m=+16.986911480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret") pod "global-pull-secret-syncer-fsv7s" (UID: "30a8139c-c65b-45ea-aa17-645a0e4c634a") : object "kube-system"/"original-pull-secret" not registered
Jul 30 02:26:13.340286 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:13.340188    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r podName:5edd31cd-4288-4f9f-a1a0-a840455c53f2 nodeName:}" failed. No retries permitted until 2026-07-30 02:26:21.340153594 +0000 UTC m=+16.986936993 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dcd6r" (UniqueName: "kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r") pod "network-check-target-w6w7c" (UID: "5edd31cd-4288-4f9f-a1a0-a840455c53f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 02:26:13.440648 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:13.440610    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs\") pod \"network-metrics-daemon-wphj8\" (UID: \"ec397db7-4d16-4a1d-9ace-fb4de6e17039\") " pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:13.440817 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:13.440782    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 02:26:13.440889 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:13.440849    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs podName:ec397db7-4d16-4a1d-9ace-fb4de6e17039 nodeName:}" failed. No retries permitted until 2026-07-30 02:26:21.440830036 +0000 UTC m=+17.087613421 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs") pod "network-metrics-daemon-wphj8" (UID: "ec397db7-4d16-4a1d-9ace-fb4de6e17039") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 02:26:13.794697 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:13.794622    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:13.794856 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:13.794752    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsv7s" podUID="30a8139c-c65b-45ea-aa17-645a0e4c634a"
Jul 30 02:26:13.795147 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:13.795107    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:13.795255 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:13.795220    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6w7c" podUID="5edd31cd-4288-4f9f-a1a0-a840455c53f2"
Jul 30 02:26:14.795404 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:14.795370    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:14.795758 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:14.795469    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:26:15.794771 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:15.794692    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:15.794771 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:15.794731    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:15.795003 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:15.794818    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsv7s" podUID="30a8139c-c65b-45ea-aa17-645a0e4c634a"
Jul 30 02:26:15.795003 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:15.794923    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6w7c" podUID="5edd31cd-4288-4f9f-a1a0-a840455c53f2"
Jul 30 02:26:16.794631 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:16.794598    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:16.795076 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:16.794731    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:26:17.794526 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:17.794494    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:17.794707 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:17.794499    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:17.794707 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:17.794591    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsv7s" podUID="30a8139c-c65b-45ea-aa17-645a0e4c634a"
Jul 30 02:26:17.794707 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:17.794673    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6w7c" podUID="5edd31cd-4288-4f9f-a1a0-a840455c53f2"
Jul 30 02:26:18.794301 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:18.794221    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:18.794455 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:18.794353    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:26:19.794790 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:19.794757    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:19.795238 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:19.794757    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:19.795238 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:19.794875    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsv7s" podUID="30a8139c-c65b-45ea-aa17-645a0e4c634a"
Jul 30 02:26:19.795238 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:19.794962    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6w7c" podUID="5edd31cd-4288-4f9f-a1a0-a840455c53f2"
Jul 30 02:26:20.794712 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:20.794674    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:20.794917 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:20.794828    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:26:21.399460 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:21.399415    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret\") pod \"global-pull-secret-syncer-fsv7s\" (UID: \"30a8139c-c65b-45ea-aa17-645a0e4c634a\") " pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:21.399460 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:21.399466    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcd6r\" (UniqueName: \"kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r\") pod \"network-check-target-w6w7c\" (UID: \"5edd31cd-4288-4f9f-a1a0-a840455c53f2\") " pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:21.399743 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:21.399590    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 02:26:21.399743 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:21.399611    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 02:26:21.399743 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:21.399626    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 02:26:21.399743 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:21.399635    2565 projected.go:196] Error preparing data for projected volume kube-api-access-dcd6r for pod openshift-network-diagnostics/network-check-target-w6w7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 02:26:21.399743 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:21.399652    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret podName:30a8139c-c65b-45ea-aa17-645a0e4c634a nodeName:}" failed. No retries permitted until 2026-07-30 02:26:37.399635946 +0000 UTC m=+33.046419329 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret") pod "global-pull-secret-syncer-fsv7s" (UID: "30a8139c-c65b-45ea-aa17-645a0e4c634a") : object "kube-system"/"original-pull-secret" not registered
Jul 30 02:26:21.399743 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:21.399678    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r podName:5edd31cd-4288-4f9f-a1a0-a840455c53f2 nodeName:}" failed. No retries permitted until 2026-07-30 02:26:37.399667352 +0000 UTC m=+33.046450736 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dcd6r" (UniqueName: "kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r") pod "network-check-target-w6w7c" (UID: "5edd31cd-4288-4f9f-a1a0-a840455c53f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 02:26:21.499881 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:21.499843    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs\") pod \"network-metrics-daemon-wphj8\" (UID: \"ec397db7-4d16-4a1d-9ace-fb4de6e17039\") " pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:21.500027 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:21.500002    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 02:26:21.500072 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:21.500056    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs podName:ec397db7-4d16-4a1d-9ace-fb4de6e17039 nodeName:}" failed. No retries permitted until 2026-07-30 02:26:37.500036863 +0000 UTC m=+33.146820247 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs") pod "network-metrics-daemon-wphj8" (UID: "ec397db7-4d16-4a1d-9ace-fb4de6e17039") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 02:26:21.794737 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:21.794649    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:21.794737 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:21.794697    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:21.795317 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:21.794807    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsv7s" podUID="30a8139c-c65b-45ea-aa17-645a0e4c634a"
Jul 30 02:26:21.795317 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:21.794943    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6w7c" podUID="5edd31cd-4288-4f9f-a1a0-a840455c53f2"
Jul 30 02:26:22.794759 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:22.794676    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:22.794945 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:22.794821    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:26:23.795091 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:23.795049    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:23.795565 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:23.795049    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:23.795565 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:23.795214    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsv7s" podUID="30a8139c-c65b-45ea-aa17-645a0e4c634a"
Jul 30 02:26:23.795565 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:23.795326    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6w7c" podUID="5edd31cd-4288-4f9f-a1a0-a840455c53f2"
Jul 30 02:26:24.795858 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:24.795817    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:24.796301 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:24.795956    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:26:25.795003 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:25.794960    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:25.795219 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:25.795012    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:25.795219 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:25.795093    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsv7s" podUID="30a8139c-c65b-45ea-aa17-645a0e4c634a"
Jul 30 02:26:25.795359 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:25.795223    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6w7c" podUID="5edd31cd-4288-4f9f-a1a0-a840455c53f2"
Jul 30 02:26:26.796510 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:26.796312    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:26.796849 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:26.796624    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:26:26.903879 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:26.903676    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-89.ec2.internal" event={"ID":"2615c8560d8f795d489b67609c7125ee","Type":"ContainerStarted","Data":"fca4e474a19819bcadd8ca44917d452537f5f01db028451b3906dfab4bb46493"}
Jul 30 02:26:26.905941 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:26.905901    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-77n5d" event={"ID":"fc2726a6-0c55-49f0-823e-58839b890c37","Type":"ContainerStarted","Data":"e00d656d6f2a948cb21c33b32d2db16bf5373cea49664dba141bad9c8d277bb4"}
Jul 30 02:26:26.909059 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:26.908881    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8" event={"ID":"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b","Type":"ContainerStarted","Data":"09f38c29fb95371e51cbb48793191467527fdbe0af80aa3ef3021e334721fc3c"}
Jul 30 02:26:26.909059 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:26.908964    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8" event={"ID":"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b","Type":"ContainerStarted","Data":"96bc493412cc9b201dde975928a152ae9f801790cea6c0c74f2b27bcc3099877"}
Jul 30 02:26:26.911307 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:26.910835    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gjr85" event={"ID":"e5d1eddb-bec6-4468-9218-e850a9214f81","Type":"ContainerStarted","Data":"62633739ece9703fa2def903de944c998f5209575869ee5a7b667a05fc94ac33"}
Jul 30 02:26:26.935325 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:26.935249    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-89.ec2.internal" podStartSLOduration=21.935226565 podStartE2EDuration="21.935226565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 02:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:26:26.917281056 +0000 UTC m=+22.564064458" watchObservedRunningTime="2026-07-30 02:26:26.935226565 +0000 UTC m=+22.582009972"
Jul 30 02:26:26.935953 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:26.935913    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gjr85" podStartSLOduration=1.317261972 podStartE2EDuration="21.935899382s" totalImagesPullingTime="20.61863741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 02:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:26:26.933532383 +0000 UTC m=+22.580315789" watchObservedRunningTime="2026-07-30 02:26:26.935899382 +0000 UTC m=+22.582682789"
Jul 30 02:26:26.953919 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:26.952929    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-77n5d" podStartSLOduration=1.4001898210000001 podStartE2EDuration="21.952910791s" totalImagesPullingTime="20.55272097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 02:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:26:26.95264427 +0000 UTC m=+22.599427676" watchObservedRunningTime="2026-07-30 02:26:26.952910791 +0000 UTC m=+22.599694196"
Jul 30 02:26:27.795058 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:27.795024    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:27.795231 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:27.795068    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:27.795231 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:27.795148    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsv7s" podUID="30a8139c-c65b-45ea-aa17-645a0e4c634a"
Jul 30 02:26:27.795352 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:27.795262    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6w7c" podUID="5edd31cd-4288-4f9f-a1a0-a840455c53f2"
Jul 30 02:26:27.914276 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:27.914196    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6" event={"ID":"228a0670-c32b-4bf3-a593-4e900bae075b","Type":"ContainerStarted","Data":"407a55df5a21fceb558bd3943b48dbe5e36ed7347f67456dc8ebb18ac4cce9cc"}
Jul 30 02:26:27.915545 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:27.915525    2565 generic.go:356] "Generic (PLEG): container finished" podID="b34ac4fa-e29d-4585-80fe-928a134067aa" containerID="bf26d12902ce4d7dde930628194bb19178e107049ed22d4ba7912a9dd49722ff" exitCode=0
Jul 30 02:26:27.915625 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:27.915596    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-278lc" event={"ID":"b34ac4fa-e29d-4585-80fe-928a134067aa","Type":"ContainerDied","Data":"bf26d12902ce4d7dde930628194bb19178e107049ed22d4ba7912a9dd49722ff"}
Jul 30 02:26:27.917352 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:27.917021    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bspvc" event={"ID":"1f1a6683-1bcf-48ce-a20b-525984e9a23a","Type":"ContainerStarted","Data":"c37537e6d6a2c39267703e5bd68178635fa0e8468a7e7f9aa20a3cc9258d2d96"}
Jul 30 02:26:27.918499 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:27.918476    2565 generic.go:356] "Generic (PLEG): container finished" podID="74d948a8b8f6069f9ae71aaf0197e360" containerID="b83e85215b67a66bb517c18722c171e88803a1012ecb7c3eb2f600540ba6105a" exitCode=0
Jul 30 02:26:27.918592 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:27.918549    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal" event={"ID":"74d948a8b8f6069f9ae71aaf0197e360","Type":"ContainerDied","Data":"b83e85215b67a66bb517c18722c171e88803a1012ecb7c3eb2f600540ba6105a"}
Jul 30 02:26:27.919984 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:27.919965    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bzccb" event={"ID":"648853ed-378d-4c9d-82e8-5cbb958399db","Type":"ContainerStarted","Data":"ee05a642cf6e0d689deaa38ae456035a9e52cf74f0da2965478784748bec359e"}
Jul 30 02:26:27.921275 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:27.921255    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b5trq" event={"ID":"a7211fce-5509-4ca2-8fb1-33a73122baec","Type":"ContainerStarted","Data":"9c033d2b722640586299a03e9cc718b74c4c1f375d891a9567345f830c476fc7"}
Jul 30 02:26:27.924013 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:27.923992    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8" event={"ID":"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b","Type":"ContainerStarted","Data":"dbfefbe17684c8e0e249fc5566adca55a6b5ba7fcfc372cd7b5b2752dbc7ac77"}
Jul 30 02:26:27.924087 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:27.924061    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8" event={"ID":"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b","Type":"ContainerStarted","Data":"51fa409eec1ef64be67e807ed98af91bd65fe2c5bfa45a9fb2db7e8df4b02fcb"}
Jul 30 02:26:27.924125 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:27.924099    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8" event={"ID":"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b","Type":"ContainerStarted","Data":"ee93703afebf008fefd9de21a12cdba79c9e0b6242679e3fc1a41c0ef92ced65"}
Jul 30 02:26:27.924161 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:27.924151    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8" event={"ID":"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b","Type":"ContainerStarted","Data":"6fc071077257052691b07cc774d59f78aa95b00bab1c9e6c914b320e83fcf34c"}
Jul 30 02:26:27.960689 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:27.960655    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bzccb" podStartSLOduration=2.426813642 podStartE2EDuration="22.960641504s" totalImagesPullingTime="20.533827862s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 02:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:26:27.960440483 +0000 UTC m=+23.607223887" watchObservedRunningTime="2026-07-30 02:26:27.960641504 +0000 UTC m=+23.607424905"
Jul 30 02:26:27.974987 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:27.974941    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bspvc" podStartSLOduration=6.700162936 podStartE2EDuration="22.974924459s" totalImagesPullingTime="16.274761523s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 02:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:26:27.97485952 +0000 UTC m=+23.621642925" watchObservedRunningTime="2026-07-30 02:26:27.974924459 +0000 UTC m=+23.621707864"
Jul 30 02:26:28.002691 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:28.002507    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b5trq" podStartSLOduration=2.463786808 podStartE2EDuration="23.002487642s" totalImagesPullingTime="20.538700834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:26:28.002393013 +0000 UTC m=+23.649176418" watchObservedRunningTime="2026-07-30 02:26:28.002487642 +0000 UTC m=+23.649271043"
Jul 30 02:26:28.259880 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:28.259859    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 30 02:26:28.730537 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:28.730438    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-30T02:26:28.259875538Z","UUID":"345319ae-91c2-4620-9505-10e9128a3070","Handler":null,"Name":"","Endpoint":""}
Jul 30 02:26:28.733368 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:28.733340    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 30 02:26:28.733517 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:28.733385    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 30 02:26:28.794280 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:28.794252    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:28.794532 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:28.794397    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:26:28.929765 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:28.929721    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal" event={"ID":"74d948a8b8f6069f9ae71aaf0197e360","Type":"ContainerStarted","Data":"65a46ec4bca679ee32f643b769f86652cf70470031bfd15de0806b99c4621f2a"}
Jul 30 02:26:28.932087 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:28.932057    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9hzfd" event={"ID":"b8143ec7-9638-4e96-87eb-efe88a8ca5ec","Type":"ContainerStarted","Data":"4296d0bc0443c3ef50ac83e766a2bd034d8ed47bfef51dbded576cd4e060f8dc"}
Jul 30 02:26:28.934249 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:28.934191    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6" event={"ID":"228a0670-c32b-4bf3-a593-4e900bae075b","Type":"ContainerStarted","Data":"433c04e2ff42ae527d5f1349bd254d8614bf75bebd226fec11f080cb07b30853"}
Jul 30 02:26:28.944816 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:28.944770    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-89.ec2.internal" podStartSLOduration=23.944757568 podStartE2EDuration="23.944757568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 02:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:26:28.944532175 +0000 UTC m=+24.591315581" watchObservedRunningTime="2026-07-30 02:26:28.944757568 +0000 UTC m=+24.591540974"
Jul 30 02:26:28.959836 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:28.959797    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9hzfd" podStartSLOduration=3.392974869 podStartE2EDuration="23.959784771s" totalImagesPullingTime="20.566809902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 02:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:26:28.959594383 +0000 UTC m=+24.606377787" watchObservedRunningTime="2026-07-30 02:26:28.959784771 +0000 UTC m=+24.606568212"
Jul 30 02:26:29.341572 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:29.341541    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b5trq"
Jul 30 02:26:29.794334 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:29.794307    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:29.794334 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:29.794329    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:29.794515 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:29.794424    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsv7s" podUID="30a8139c-c65b-45ea-aa17-645a0e4c634a"
Jul 30 02:26:29.794579 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:29.794558    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6w7c" podUID="5edd31cd-4288-4f9f-a1a0-a840455c53f2"
Jul 30 02:26:29.941235 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:29.941119    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8" event={"ID":"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b","Type":"ContainerStarted","Data":"f9c3e96446b578bf500df45bb6992bd503d01162e4a2529fa0d67b88b0455a15"}
Jul 30 02:26:29.943677 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:29.943645    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6" event={"ID":"228a0670-c32b-4bf3-a593-4e900bae075b","Type":"ContainerStarted","Data":"9f0d7182d146cffc0fe0d0dd6a3124a240eef6684803869a43a71650bac2b274"}
Jul 30 02:26:29.961947 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:29.961897    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phsz6" podStartSLOduration=1.860832339 podStartE2EDuration="24.961877015s" totalImagesPullingTime="23.101044676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 02:26:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:26:29.961516788 +0000 UTC m=+25.608300194" watchObservedRunningTime="2026-07-30 02:26:29.961877015 +0000 UTC m=+25.608660424"
Jul 30 02:26:30.027075 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:30.027043    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b5trq"
Jul 30 02:26:30.027670 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:30.027645    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b5trq"
Jul 30 02:26:30.794632 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:30.794597    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:30.794957 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:30.794741    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:26:30.946633 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:30.946606    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b5trq"
Jul 30 02:26:31.794789 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:31.794600    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:31.794789 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:31.794667    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:31.794789 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:31.794752    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsv7s" podUID="30a8139c-c65b-45ea-aa17-645a0e4c634a"
Jul 30 02:26:31.795038 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:31.794874    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6w7c" podUID="5edd31cd-4288-4f9f-a1a0-a840455c53f2"
Jul 30 02:26:32.794448 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:32.794247    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:32.794946 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:32.794505    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:26:32.954838 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:32.954801    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8" event={"ID":"3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b","Type":"ContainerStarted","Data":"e0854f3b846b525619c691390ac305260ffd619fc79c7e958f01b2e2240c34f5"}
Jul 30 02:26:32.955041 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:32.955019    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:32.955106 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:32.955095    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:32.956865 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:32.956835    2565 generic.go:356] "Generic (PLEG): container finished" podID="b34ac4fa-e29d-4585-80fe-928a134067aa" containerID="827ed32705b2d758dea81017f0abc5b03799a0841d1bfa02a8a6b80ce89ac8ca" exitCode=0
Jul 30 02:26:32.956984 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:32.956916    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-278lc" event={"ID":"b34ac4fa-e29d-4585-80fe-928a134067aa","Type":"ContainerDied","Data":"827ed32705b2d758dea81017f0abc5b03799a0841d1bfa02a8a6b80ce89ac8ca"}
Jul 30 02:26:32.972587 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:32.972561    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:32.981594 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:32.981558    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8" podStartSLOduration=7.357046187 podStartE2EDuration="27.981533614s" totalImagesPullingTime="20.624487427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 02:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:26:32.981314266 +0000 UTC m=+28.628097674" watchObservedRunningTime="2026-07-30 02:26:32.981533614 +0000 UTC m=+28.628317021"
Jul 30 02:26:33.794676 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:33.794647    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:33.795421 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:33.794647    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:33.795421 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:33.794751    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6w7c" podUID="5edd31cd-4288-4f9f-a1a0-a840455c53f2"
Jul 30 02:26:33.795421 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:33.794821    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsv7s" podUID="30a8139c-c65b-45ea-aa17-645a0e4c634a"
Jul 30 02:26:33.960019 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:33.959997    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:33.977794 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:33.977766    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:26:34.261191 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:34.261140    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fsv7s"]
Jul 30 02:26:34.261368 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:34.261260    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:34.261368 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:34.261347    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsv7s" podUID="30a8139c-c65b-45ea-aa17-645a0e4c634a"
Jul 30 02:26:34.264589 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:34.264562    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w6w7c"]
Jul 30 02:26:34.264703 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:34.264662    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:34.264809 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:34.264767    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6w7c" podUID="5edd31cd-4288-4f9f-a1a0-a840455c53f2"
Jul 30 02:26:34.265274 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:34.265250    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wphj8"]
Jul 30 02:26:34.265395 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:34.265355    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:34.265471 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:34.265454    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:26:34.963562 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:34.963527    2565 generic.go:356] "Generic (PLEG): container finished" podID="b34ac4fa-e29d-4585-80fe-928a134067aa" containerID="f672045a039582ad04aa87d8733b97283d75ad8abdd3c9a5ab70d299f5a8b94a" exitCode=0
Jul 30 02:26:34.963993 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:34.963608    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-278lc" event={"ID":"b34ac4fa-e29d-4585-80fe-928a134067aa","Type":"ContainerDied","Data":"f672045a039582ad04aa87d8733b97283d75ad8abdd3c9a5ab70d299f5a8b94a"}
Jul 30 02:26:35.795232 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:35.795197    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:35.795232 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:35.795232    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:35.795528 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:35.795205    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:35.795528 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:35.795343    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:26:35.795528 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:35.795399    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsv7s" podUID="30a8139c-c65b-45ea-aa17-645a0e4c634a"
Jul 30 02:26:35.795528 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:35.795461    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6w7c" podUID="5edd31cd-4288-4f9f-a1a0-a840455c53f2"
Jul 30 02:26:36.972780 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:36.972743    2565 generic.go:356] "Generic (PLEG): container finished" podID="b34ac4fa-e29d-4585-80fe-928a134067aa" containerID="f68ee2478ee55bb9111fa9b4dd8955c827b65c2bf4f7712d5d37fd695cc25c3b" exitCode=0
Jul 30 02:26:36.973243 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:36.972794    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-278lc" event={"ID":"b34ac4fa-e29d-4585-80fe-928a134067aa","Type":"ContainerDied","Data":"f68ee2478ee55bb9111fa9b4dd8955c827b65c2bf4f7712d5d37fd695cc25c3b"}
Jul 30 02:26:37.424320 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:37.424281    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcd6r\" (UniqueName: \"kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r\") pod \"network-check-target-w6w7c\" (UID: \"5edd31cd-4288-4f9f-a1a0-a840455c53f2\") " pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:37.424547 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:37.424352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret\") pod \"global-pull-secret-syncer-fsv7s\" (UID: \"30a8139c-c65b-45ea-aa17-645a0e4c634a\") " pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:37.424547 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:37.424436    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 02:26:37.424547 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:37.424470    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 02:26:37.424547 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:37.424486    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret podName:30a8139c-c65b-45ea-aa17-645a0e4c634a nodeName:}" failed. No retries permitted until 2026-07-30 02:27:09.424473812 +0000 UTC m=+65.071257196 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret") pod "global-pull-secret-syncer-fsv7s" (UID: "30a8139c-c65b-45ea-aa17-645a0e4c634a") : object "kube-system"/"original-pull-secret" not registered
Jul 30 02:26:37.424547 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:37.424495    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 02:26:37.424547 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:37.424511    2565 projected.go:196] Error preparing data for projected volume kube-api-access-dcd6r for pod openshift-network-diagnostics/network-check-target-w6w7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 02:26:37.424759 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:37.424560    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r podName:5edd31cd-4288-4f9f-a1a0-a840455c53f2 nodeName:}" failed. No retries permitted until 2026-07-30 02:27:09.424543354 +0000 UTC m=+65.071326743 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dcd6r" (UniqueName: "kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r") pod "network-check-target-w6w7c" (UID: "5edd31cd-4288-4f9f-a1a0-a840455c53f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 02:26:37.524790 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:37.524755    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs\") pod \"network-metrics-daemon-wphj8\" (UID: \"ec397db7-4d16-4a1d-9ace-fb4de6e17039\") " pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:37.524936 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:37.524900    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 02:26:37.524980 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:37.524968    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs podName:ec397db7-4d16-4a1d-9ace-fb4de6e17039 nodeName:}" failed. No retries permitted until 2026-07-30 02:27:09.524945354 +0000 UTC m=+65.171728738 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs") pod "network-metrics-daemon-wphj8" (UID: "ec397db7-4d16-4a1d-9ace-fb4de6e17039") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 02:26:37.795077 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:37.795044    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:37.795286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:37.795095    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:37.795286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:37.795095    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:37.795286 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:37.795191    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsv7s" podUID="30a8139c-c65b-45ea-aa17-645a0e4c634a"
Jul 30 02:26:37.795286 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:37.795266    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6w7c" podUID="5edd31cd-4288-4f9f-a1a0-a840455c53f2"
Jul 30 02:26:37.795494 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:37.795347    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:26:39.794197 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:39.794145    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:39.794576 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:39.794153    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:39.794576 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:39.794288    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w6w7c" podUID="5edd31cd-4288-4f9f-a1a0-a840455c53f2"
Jul 30 02:26:39.794576 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:39.794154    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:39.794576 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:39.794381    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:26:39.794576 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:39.794437    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fsv7s" podUID="30a8139c-c65b-45ea-aa17-645a0e4c634a"
Jul 30 02:26:40.720563 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.720530    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-89.ec2.internal" event="NodeReady"
Jul 30 02:26:40.720744 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.720697    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 30 02:26:40.762213 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.762162    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7rhjt"]
Jul 30 02:26:40.767296 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.767276    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8gs9b"]
Jul 30 02:26:40.767465 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.767442    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:26:40.769993 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.769966    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 30 02:26:40.770161 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.770013    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 30 02:26:40.770161 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.770027    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5sqnz\""
Jul 30 02:26:40.770650 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.770635    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8gs9b"
Jul 30 02:26:40.772925 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.772909    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 30 02:26:40.773055 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.773028    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 30 02:26:40.773289 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.773273    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tg8zh\""
Jul 30 02:26:40.773720 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.773557    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 30 02:26:40.775958 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.775380    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7rhjt"]
Jul 30 02:26:40.777982 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.777958    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8gs9b"]
Jul 30 02:26:40.850349 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.850316    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6641518e-6c07-41e1-92b0-470be0e94ddc-config-volume\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:26:40.850762 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.850373    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4dfc729-47bc-41b7-b64d-0c1b45010108-cert\") pod \"ingress-canary-8gs9b\" (UID: \"d4dfc729-47bc-41b7-b64d-0c1b45010108\") " pod="openshift-ingress-canary/ingress-canary-8gs9b"
Jul 30 02:26:40.850762 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.850454    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26gf9\" (UniqueName: \"kubernetes.io/projected/6641518e-6c07-41e1-92b0-470be0e94ddc-kube-api-access-26gf9\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:26:40.850762 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.850534    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5q8x\" (UniqueName: \"kubernetes.io/projected/d4dfc729-47bc-41b7-b64d-0c1b45010108-kube-api-access-r5q8x\") pod \"ingress-canary-8gs9b\" (UID: \"d4dfc729-47bc-41b7-b64d-0c1b45010108\") " pod="openshift-ingress-canary/ingress-canary-8gs9b"
Jul 30 02:26:40.850762 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.850563    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:26:40.850762 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.850593    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6641518e-6c07-41e1-92b0-470be0e94ddc-tmp-dir\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:26:40.951557 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.951527    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4dfc729-47bc-41b7-b64d-0c1b45010108-cert\") pod \"ingress-canary-8gs9b\" (UID: \"d4dfc729-47bc-41b7-b64d-0c1b45010108\") " pod="openshift-ingress-canary/ingress-canary-8gs9b"
Jul 30 02:26:40.951742 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.951581    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26gf9\" (UniqueName: \"kubernetes.io/projected/6641518e-6c07-41e1-92b0-470be0e94ddc-kube-api-access-26gf9\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:26:40.951742 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.951624    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5q8x\" (UniqueName: \"kubernetes.io/projected/d4dfc729-47bc-41b7-b64d-0c1b45010108-kube-api-access-r5q8x\") pod \"ingress-canary-8gs9b\" (UID: \"d4dfc729-47bc-41b7-b64d-0c1b45010108\") " pod="openshift-ingress-canary/ingress-canary-8gs9b"
Jul 30 02:26:40.951742 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.951650    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:26:40.951742 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.951669    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6641518e-6c07-41e1-92b0-470be0e94ddc-tmp-dir\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:26:40.951742 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.951691    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6641518e-6c07-41e1-92b0-470be0e94ddc-config-volume\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:26:40.952495 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:40.952255    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 02:26:40.952495 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.952315    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6641518e-6c07-41e1-92b0-470be0e94ddc-config-volume\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:26:40.952495 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:40.952326    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls podName:6641518e-6c07-41e1-92b0-470be0e94ddc nodeName:}" failed. No retries permitted until 2026-07-30 02:26:41.452303985 +0000 UTC m=+37.099087368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls") pod "dns-default-7rhjt" (UID: "6641518e-6c07-41e1-92b0-470be0e94ddc") : secret "dns-default-metrics-tls" not found
Jul 30 02:26:40.952733 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.952585    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6641518e-6c07-41e1-92b0-470be0e94ddc-tmp-dir\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:26:40.957023 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.956999    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4dfc729-47bc-41b7-b64d-0c1b45010108-cert\") pod \"ingress-canary-8gs9b\" (UID: \"d4dfc729-47bc-41b7-b64d-0c1b45010108\") " pod="openshift-ingress-canary/ingress-canary-8gs9b"
Jul 30 02:26:40.961556 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.961527    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26gf9\" (UniqueName: \"kubernetes.io/projected/6641518e-6c07-41e1-92b0-470be0e94ddc-kube-api-access-26gf9\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:26:40.961772 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:40.961755    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5q8x\" (UniqueName: \"kubernetes.io/projected/d4dfc729-47bc-41b7-b64d-0c1b45010108-kube-api-access-r5q8x\") pod \"ingress-canary-8gs9b\" (UID: \"d4dfc729-47bc-41b7-b64d-0c1b45010108\") " pod="openshift-ingress-canary/ingress-canary-8gs9b"
Jul 30 02:26:41.088949 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:41.088755    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8gs9b"
Jul 30 02:26:41.231262 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:41.231226    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8gs9b"]
Jul 30 02:26:41.234213 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:26:41.234162    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4dfc729_47bc_41b7_b64d_0c1b45010108.slice/crio-4957fe08cb9bb14b9e98ecbef40d8f1f3c32238b2d0e84ae6620d06003c9e68c WatchSource:0}: Error finding container 4957fe08cb9bb14b9e98ecbef40d8f1f3c32238b2d0e84ae6620d06003c9e68c: Status 404 returned error can't find the container with id 4957fe08cb9bb14b9e98ecbef40d8f1f3c32238b2d0e84ae6620d06003c9e68c
Jul 30 02:26:41.455859 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:41.455787    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:26:41.455979 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:41.455921    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 02:26:41.456018 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:41.455978    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls podName:6641518e-6c07-41e1-92b0-470be0e94ddc nodeName:}" failed. No retries permitted until 2026-07-30 02:26:42.455961994 +0000 UTC m=+38.102745381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls") pod "dns-default-7rhjt" (UID: "6641518e-6c07-41e1-92b0-470be0e94ddc") : secret "dns-default-metrics-tls" not found
Jul 30 02:26:41.794969 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:41.794934    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:26:41.795186 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:41.795064    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:26:41.795186 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:41.794944    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:26:41.798370 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:41.798351    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 30 02:26:41.799665 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:41.799488    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 30 02:26:41.799665 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:41.799499    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vvjjn\""
Jul 30 02:26:41.799665 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:41.799536    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-67lr8\""
Jul 30 02:26:41.799665 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:41.799550    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 30 02:26:41.799665 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:41.799643    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 30 02:26:41.987156 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:41.987085    2565 generic.go:356] "Generic (PLEG): container finished" podID="b34ac4fa-e29d-4585-80fe-928a134067aa" containerID="8cba24a7f90f35a949b430f474c58d88d40a3a6ce104950fbef0d188145cc13f" exitCode=0
Jul 30 02:26:41.987905 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:41.987201    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-278lc" event={"ID":"b34ac4fa-e29d-4585-80fe-928a134067aa","Type":"ContainerDied","Data":"8cba24a7f90f35a949b430f474c58d88d40a3a6ce104950fbef0d188145cc13f"}
Jul 30 02:26:41.988431 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:41.988402    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8gs9b" event={"ID":"d4dfc729-47bc-41b7-b64d-0c1b45010108","Type":"ContainerStarted","Data":"4957fe08cb9bb14b9e98ecbef40d8f1f3c32238b2d0e84ae6620d06003c9e68c"}
Jul 30 02:26:42.463890 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:42.463798    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:26:42.464041 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:42.463966    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 02:26:42.464041 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:42.464039    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls podName:6641518e-6c07-41e1-92b0-470be0e94ddc nodeName:}" failed. No retries permitted until 2026-07-30 02:26:44.464022996 +0000 UTC m=+40.110806379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls") pod "dns-default-7rhjt" (UID: "6641518e-6c07-41e1-92b0-470be0e94ddc") : secret "dns-default-metrics-tls" not found
Jul 30 02:26:42.995324 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:42.995126    2565 generic.go:356] "Generic (PLEG): container finished" podID="b34ac4fa-e29d-4585-80fe-928a134067aa" containerID="1f0e66c352df93d6c62963c99815b3debcdc926e8804019bea7122f20be82e15" exitCode=0
Jul 30 02:26:42.995324 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:42.995204    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-278lc" event={"ID":"b34ac4fa-e29d-4585-80fe-928a134067aa","Type":"ContainerDied","Data":"1f0e66c352df93d6c62963c99815b3debcdc926e8804019bea7122f20be82e15"}
Jul 30 02:26:44.000632 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:44.000541    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-278lc" event={"ID":"b34ac4fa-e29d-4585-80fe-928a134067aa","Type":"ContainerStarted","Data":"484b9d371ff4a107e021f97c6e130b14e9c183b025eeab3a3367c2673aa91620"}
Jul 30 02:26:44.001988 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:44.001961    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8gs9b" event={"ID":"d4dfc729-47bc-41b7-b64d-0c1b45010108","Type":"ContainerStarted","Data":"86494fb7166f297d78f5dcb57ed8360443e191a3097cc566cc044b53e03f08b3"}
Jul 30 02:26:44.025911 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:44.025865    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-278lc" podStartSLOduration=8.083000459 podStartE2EDuration="39.025846166s" totalImagesPullingTime="30.942845707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 02:26:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:26:44.0246788 +0000 UTC m=+39.671462206" watchObservedRunningTime="2026-07-30 02:26:44.025846166 +0000 UTC m=+39.672629567"
Jul 30 02:26:44.040247 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:44.040205    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8gs9b" podStartSLOduration=1.561167937 podStartE2EDuration="4.04019166s" totalImagesPullingTime="2.479023723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:26:44.039666971 +0000 UTC m=+39.686450375" watchObservedRunningTime="2026-07-30 02:26:44.04019166 +0000 UTC m=+39.686975059"
Jul 30 02:26:44.478947 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:44.478913    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:26:44.479128 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:44.479022    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 02:26:44.479128 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:44.479072    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls podName:6641518e-6c07-41e1-92b0-470be0e94ddc nodeName:}" failed. No retries permitted until 2026-07-30 02:26:48.479059449 +0000 UTC m=+44.125842831 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls") pod "dns-default-7rhjt" (UID: "6641518e-6c07-41e1-92b0-470be0e94ddc") : secret "dns-default-metrics-tls" not found
Jul 30 02:26:48.507058 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:48.507012    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:26:48.507463 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:48.507128    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 02:26:48.507463 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:48.507195    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls podName:6641518e-6c07-41e1-92b0-470be0e94ddc nodeName:}" failed. No retries permitted until 2026-07-30 02:26:56.507182189 +0000 UTC m=+52.153965585 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls") pod "dns-default-7rhjt" (UID: "6641518e-6c07-41e1-92b0-470be0e94ddc") : secret "dns-default-metrics-tls" not found
Jul 30 02:26:56.560550 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:26:56.560509    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:26:56.561001 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:56.560639    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 02:26:56.561001 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:26:56.560707    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls podName:6641518e-6c07-41e1-92b0-470be0e94ddc nodeName:}" failed. No retries permitted until 2026-07-30 02:27:12.56069032 +0000 UTC m=+68.207473703 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls") pod "dns-default-7rhjt" (UID: "6641518e-6c07-41e1-92b0-470be0e94ddc") : secret "dns-default-metrics-tls" not found
Jul 30 02:27:05.985572 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:05.985538    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rrn8"
Jul 30 02:27:09.449418 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:09.449369    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcd6r\" (UniqueName: \"kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r\") pod \"network-check-target-w6w7c\" (UID: \"5edd31cd-4288-4f9f-a1a0-a840455c53f2\") " pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:27:09.449897 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:09.449455    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret\") pod \"global-pull-secret-syncer-fsv7s\" (UID: \"30a8139c-c65b-45ea-aa17-645a0e4c634a\") " pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:27:09.452688 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:09.452671    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 30 02:27:09.452757 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:09.452671    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 30 02:27:09.462529 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:09.462507    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 30 02:27:09.463518 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:09.463500    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/30a8139c-c65b-45ea-aa17-645a0e4c634a-original-pull-secret\") pod \"global-pull-secret-syncer-fsv7s\" (UID: \"30a8139c-c65b-45ea-aa17-645a0e4c634a\") " pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:27:09.472934 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:09.472911    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcd6r\" (UniqueName: \"kubernetes.io/projected/5edd31cd-4288-4f9f-a1a0-a840455c53f2-kube-api-access-dcd6r\") pod \"network-check-target-w6w7c\" (UID: \"5edd31cd-4288-4f9f-a1a0-a840455c53f2\") " pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:27:09.550005 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:09.549975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs\") pod \"network-metrics-daemon-wphj8\" (UID: \"ec397db7-4d16-4a1d-9ace-fb4de6e17039\") " pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:27:09.553285 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:09.553268    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 30 02:27:09.560372 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:27:09.560357    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 30 02:27:09.560460 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:27:09.560410    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs podName:ec397db7-4d16-4a1d-9ace-fb4de6e17039 nodeName:}" failed. No retries permitted until 2026-07-30 02:28:13.560395007 +0000 UTC m=+129.207178389 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs") pod "network-metrics-daemon-wphj8" (UID: "ec397db7-4d16-4a1d-9ace-fb4de6e17039") : secret "metrics-daemon-secret" not found
Jul 30 02:27:09.716325 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:09.716241    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-67lr8\""
Jul 30 02:27:09.719379 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:09.719364    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fsv7s"
Jul 30 02:27:09.724832 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:09.724814    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:27:09.849652 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:09.849613    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fsv7s"]
Jul 30 02:27:09.852545 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:27:09.852521    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a8139c_c65b_45ea_aa17_645a0e4c634a.slice/crio-909f12d190bb816aba6fbf80f1a3aa44cdb413046f0285080e6bf9f14f4bd15f WatchSource:0}: Error finding container 909f12d190bb816aba6fbf80f1a3aa44cdb413046f0285080e6bf9f14f4bd15f: Status 404 returned error can't find the container with id 909f12d190bb816aba6fbf80f1a3aa44cdb413046f0285080e6bf9f14f4bd15f
Jul 30 02:27:09.864288 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:09.864270    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w6w7c"]
Jul 30 02:27:09.866197 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:27:09.866149    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edd31cd_4288_4f9f_a1a0_a840455c53f2.slice/crio-b99acb583fa1099fbf294cb5f7f69646fb73c64a8cea9a5646b97f179837b0c9 WatchSource:0}: Error finding container b99acb583fa1099fbf294cb5f7f69646fb73c64a8cea9a5646b97f179837b0c9: Status 404 returned error can't find the container with id b99acb583fa1099fbf294cb5f7f69646fb73c64a8cea9a5646b97f179837b0c9
Jul 30 02:27:10.058943 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:10.058907    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w6w7c" event={"ID":"5edd31cd-4288-4f9f-a1a0-a840455c53f2","Type":"ContainerStarted","Data":"b99acb583fa1099fbf294cb5f7f69646fb73c64a8cea9a5646b97f179837b0c9"}
Jul 30 02:27:10.059880 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:10.059860    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fsv7s" event={"ID":"30a8139c-c65b-45ea-aa17-645a0e4c634a","Type":"ContainerStarted","Data":"909f12d190bb816aba6fbf80f1a3aa44cdb413046f0285080e6bf9f14f4bd15f"}
Jul 30 02:27:12.573345 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:12.573296    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:27:12.573810 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:27:12.573468    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 02:27:12.573810 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:27:12.573552    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls podName:6641518e-6c07-41e1-92b0-470be0e94ddc nodeName:}" failed. No retries permitted until 2026-07-30 02:27:44.573529947 +0000 UTC m=+100.220313335 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls") pod "dns-default-7rhjt" (UID: "6641518e-6c07-41e1-92b0-470be0e94ddc") : secret "dns-default-metrics-tls" not found
Jul 30 02:27:15.078647 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:15.078617    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w6w7c" event={"ID":"5edd31cd-4288-4f9f-a1a0-a840455c53f2","Type":"ContainerStarted","Data":"096a57ee2b9620b781d555c65361e9ea3693e0eaf0bc42f72d15f4678b9c8182"}
Jul 30 02:27:15.079185 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:15.078726    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:27:15.080039 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:15.080017    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fsv7s" event={"ID":"30a8139c-c65b-45ea-aa17-645a0e4c634a","Type":"ContainerStarted","Data":"aaea32b9c13d7bba6185b2c0fbd390668ab4fb69cf9426ca9c5baa12f891fe8c"}
Jul 30 02:27:15.097174 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:15.097135    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w6w7c" podStartSLOduration=65.415920004 podStartE2EDuration="1m10.09712251s" totalImagesPullingTime="4.681202506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:27:15.096867313 +0000 UTC m=+70.743650718" watchObservedRunningTime="2026-07-30 02:27:15.09712251 +0000 UTC m=+70.743905914"
Jul 30 02:27:15.111091 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:15.111058    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fsv7s" podStartSLOduration=65.404700206 podStartE2EDuration="1m10.111046139s" totalImagesPullingTime="4.706345933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 02:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:27:15.110847215 +0000 UTC m=+70.757630619" watchObservedRunningTime="2026-07-30 02:27:15.111046139 +0000 UTC m=+70.757829543"
Jul 30 02:27:44.589991 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:44.589955    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:27:44.590406 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:27:44.590103    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 02:27:44.590406 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:27:44.590190    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls podName:6641518e-6c07-41e1-92b0-470be0e94ddc nodeName:}" failed. No retries permitted until 2026-07-30 02:28:48.590155652 +0000 UTC m=+164.236939035 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls") pod "dns-default-7rhjt" (UID: "6641518e-6c07-41e1-92b0-470be0e94ddc") : secret "dns-default-metrics-tls" not found
Jul 30 02:27:46.084360 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:27:46.084328    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w6w7c"
Jul 30 02:28:13.581395 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:13.581336    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs\") pod \"network-metrics-daemon-wphj8\" (UID: \"ec397db7-4d16-4a1d-9ace-fb4de6e17039\") " pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:28:13.581875 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:28:13.581495    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 30 02:28:13.581875 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:28:13.581564    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs podName:ec397db7-4d16-4a1d-9ace-fb4de6e17039 nodeName:}" failed. No retries permitted until 2026-07-30 02:30:15.581548244 +0000 UTC m=+251.228331627 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs") pod "network-metrics-daemon-wphj8" (UID: "ec397db7-4d16-4a1d-9ace-fb4de6e17039") : secret "metrics-daemon-secret" not found
Jul 30 02:28:35.486075 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.486038    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"]
Jul 30 02:28:35.489052 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.489033    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:28:35.491577 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.491553    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 30 02:28:35.493052 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.493031    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 30 02:28:35.493193 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.493050    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 30 02:28:35.493193 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.493034    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 30 02:28:35.493193 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.493034    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 30 02:28:35.493376 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.493057    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 30 02:28:35.493376 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.493031    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 30 02:28:35.502427 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.500509    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"]
Jul 30 02:28:35.627407 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.627374    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/af4e9838-5365-4fea-aed4-3200eacf14c4-ca\") pod \"cluster-proxy-proxy-agent-76d97b5f99-hgzj7\" (UID: \"af4e9838-5365-4fea-aed4-3200eacf14c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:28:35.627575 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.627415    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af4e9838-5365-4fea-aed4-3200eacf14c4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76d97b5f99-hgzj7\" (UID: \"af4e9838-5365-4fea-aed4-3200eacf14c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:28:35.627575 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.627451    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjnd7\" (UniqueName: \"kubernetes.io/projected/af4e9838-5365-4fea-aed4-3200eacf14c4-kube-api-access-hjnd7\") pod \"cluster-proxy-proxy-agent-76d97b5f99-hgzj7\" (UID: \"af4e9838-5365-4fea-aed4-3200eacf14c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:28:35.627651 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.627565    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/af4e9838-5365-4fea-aed4-3200eacf14c4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76d97b5f99-hgzj7\" (UID: \"af4e9838-5365-4fea-aed4-3200eacf14c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:28:35.627651 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.627607    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4e9838-5365-4fea-aed4-3200eacf14c4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76d97b5f99-hgzj7\" (UID: \"af4e9838-5365-4fea-aed4-3200eacf14c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:28:35.627651 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.627639    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/af4e9838-5365-4fea-aed4-3200eacf14c4-hub\") pod \"cluster-proxy-proxy-agent-76d97b5f99-hgzj7\" (UID: \"af4e9838-5365-4fea-aed4-3200eacf14c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:28:35.728533 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.728501    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af4e9838-5365-4fea-aed4-3200eacf14c4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76d97b5f99-hgzj7\" (UID: \"af4e9838-5365-4fea-aed4-3200eacf14c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:28:35.728630 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.728556    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjnd7\" (UniqueName: \"kubernetes.io/projected/af4e9838-5365-4fea-aed4-3200eacf14c4-kube-api-access-hjnd7\") pod \"cluster-proxy-proxy-agent-76d97b5f99-hgzj7\" (UID: \"af4e9838-5365-4fea-aed4-3200eacf14c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:28:35.728630 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.728607    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/af4e9838-5365-4fea-aed4-3200eacf14c4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76d97b5f99-hgzj7\" (UID: \"af4e9838-5365-4fea-aed4-3200eacf14c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:28:35.728740 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.728641    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4e9838-5365-4fea-aed4-3200eacf14c4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76d97b5f99-hgzj7\" (UID: \"af4e9838-5365-4fea-aed4-3200eacf14c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:28:35.728740 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.728672    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/af4e9838-5365-4fea-aed4-3200eacf14c4-hub\") pod \"cluster-proxy-proxy-agent-76d97b5f99-hgzj7\" (UID: \"af4e9838-5365-4fea-aed4-3200eacf14c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:28:35.728740 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.728698    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/af4e9838-5365-4fea-aed4-3200eacf14c4-ca\") pod \"cluster-proxy-proxy-agent-76d97b5f99-hgzj7\" (UID: \"af4e9838-5365-4fea-aed4-3200eacf14c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:28:35.729426 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.729330    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4e9838-5365-4fea-aed4-3200eacf14c4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76d97b5f99-hgzj7\" (UID: \"af4e9838-5365-4fea-aed4-3200eacf14c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:28:35.731219 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.731190    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/af4e9838-5365-4fea-aed4-3200eacf14c4-ca\") pod \"cluster-proxy-proxy-agent-76d97b5f99-hgzj7\" (UID: \"af4e9838-5365-4fea-aed4-3200eacf14c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:28:35.731429 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.731406    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/af4e9838-5365-4fea-aed4-3200eacf14c4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76d97b5f99-hgzj7\" (UID: \"af4e9838-5365-4fea-aed4-3200eacf14c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:28:35.731509 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.731472    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af4e9838-5365-4fea-aed4-3200eacf14c4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76d97b5f99-hgzj7\" (UID: \"af4e9838-5365-4fea-aed4-3200eacf14c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:28:35.731509 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.731477    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/af4e9838-5365-4fea-aed4-3200eacf14c4-hub\") pod \"cluster-proxy-proxy-agent-76d97b5f99-hgzj7\" (UID: \"af4e9838-5365-4fea-aed4-3200eacf14c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:28:35.736437 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.736388    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjnd7\" (UniqueName: \"kubernetes.io/projected/af4e9838-5365-4fea-aed4-3200eacf14c4-kube-api-access-hjnd7\") pod \"cluster-proxy-proxy-agent-76d97b5f99-hgzj7\" (UID: \"af4e9838-5365-4fea-aed4-3200eacf14c4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:28:35.807188 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.807132    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:28:35.930604 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:35.930578    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"]
Jul 30 02:28:36.271443 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:36.271403    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7" event={"ID":"af4e9838-5365-4fea-aed4-3200eacf14c4","Type":"ContainerStarted","Data":"3d83956ab801ceaee9d48e65124099adbe64cdf73f25430b37ad9c84b1a0549d"}
Jul 30 02:28:40.284328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:40.284290    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7" event={"ID":"af4e9838-5365-4fea-aed4-3200eacf14c4","Type":"ContainerStarted","Data":"3bc1d74bd89273cadefc74fb15ca2cb6b380409967bfcab0f9a9f37eaa0cf83a"}
Jul 30 02:28:40.284695 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:40.284338    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7" event={"ID":"af4e9838-5365-4fea-aed4-3200eacf14c4","Type":"ContainerStarted","Data":"c880fa117d9f6cd6af0577a0f5f883b9b12a13172d5ad28c6bc0127b76671174"}
Jul 30 02:28:40.284695 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:40.284354    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7" event={"ID":"af4e9838-5365-4fea-aed4-3200eacf14c4","Type":"ContainerStarted","Data":"c2807deb012e81dd90f14b08a6687b1140675a67261c51315f3b3bef440a8b08"}
Jul 30 02:28:40.304105 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:40.304059    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7" podStartSLOduration=1.792286708 podStartE2EDuration="5.304045046s" totalImagesPullingTime="3.511758338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:28:40.303375987 +0000 UTC m=+155.950159395" watchObservedRunningTime="2026-07-30 02:28:40.304045046 +0000 UTC m=+155.950828451"
Jul 30 02:28:43.781382 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:28:43.781328    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7rhjt" podUID="6641518e-6c07-41e1-92b0-470be0e94ddc"
Jul 30 02:28:44.297047 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:44.297019    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:28:44.806595 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:28:44.806562    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:28:48.628703 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:48.628666    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:28:48.629094 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:28:48.628775    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 02:28:48.629094 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:28:48.628843    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls podName:6641518e-6c07-41e1-92b0-470be0e94ddc nodeName:}" failed. No retries permitted until 2026-07-30 02:30:50.62882871 +0000 UTC m=+286.275612093 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls") pod "dns-default-7rhjt" (UID: "6641518e-6c07-41e1-92b0-470be0e94ddc") : secret "dns-default-metrics-tls" not found
Jul 30 02:28:58.795214 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:28:58.795075    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:30:14.614239 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.614201    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9z4l6"]
Jul 30 02:30:14.617335 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.617312    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:14.620328 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.620303    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 30 02:30:14.620456 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.620340    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 30 02:30:14.620456 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.620359    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 30 02:30:14.620456 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.620341    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 30 02:30:14.621511 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.621493    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6sh2l\""
Jul 30 02:30:14.627456 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.627436    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9z4l6"]
Jul 30 02:30:14.710019 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.709987    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-data-volume\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:14.710273 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.710030    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28hwk\" (UniqueName: \"kubernetes.io/projected/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-api-access-28hwk\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:14.710273 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.710065    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:14.710273 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.710093    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-crio-socket\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:14.710273 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.710189    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:14.811361 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.811330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:14.811361 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.811367    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-data-volume\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:14.811548 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.811390    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28hwk\" (UniqueName: \"kubernetes.io/projected/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-api-access-28hwk\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:14.811548 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.811411    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:14.811548 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.811438    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-crio-socket\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:14.811548 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:14.811436    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 02:30:14.811548 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:14.811508    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-rbac-proxy-cm podName:09a07cf3-a768-4fcd-ade4-85783f4bf6d6 nodeName:}" failed. No retries permitted until 2026-07-30 02:30:15.311492568 +0000 UTC m=+250.958275950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9z4l6" (UID: "09a07cf3-a768-4fcd-ade4-85783f4bf6d6") : configmap "kube-rbac-proxy" not found
Jul 30 02:30:14.811548 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.811522    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-crio-socket\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:14.811767 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:14.811569    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 02:30:14.811767 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:14.811626    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-insights-runtime-extractor-tls podName:09a07cf3-a768-4fcd-ade4-85783f4bf6d6 nodeName:}" failed. No retries permitted until 2026-07-30 02:30:15.311612433 +0000 UTC m=+250.958395820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9z4l6" (UID: "09a07cf3-a768-4fcd-ade4-85783f4bf6d6") : secret "insights-runtime-extractor-tls" not found
Jul 30 02:30:14.811767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.811702    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-data-volume\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:14.822444 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:14.822421    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28hwk\" (UniqueName: \"kubernetes.io/projected/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-api-access-28hwk\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:15.314618 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:15.314584    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:15.314817 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:15.314643    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:15.314817 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:15.314706    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 02:30:15.314817 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:15.314751    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 02:30:15.314817 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:15.314757    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-rbac-proxy-cm podName:09a07cf3-a768-4fcd-ade4-85783f4bf6d6 nodeName:}" failed. No retries permitted until 2026-07-30 02:30:16.314742735 +0000 UTC m=+251.961526119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9z4l6" (UID: "09a07cf3-a768-4fcd-ade4-85783f4bf6d6") : configmap "kube-rbac-proxy" not found
Jul 30 02:30:15.314983 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:15.314825    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-insights-runtime-extractor-tls podName:09a07cf3-a768-4fcd-ade4-85783f4bf6d6 nodeName:}" failed. No retries permitted until 2026-07-30 02:30:16.314806902 +0000 UTC m=+251.961590286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9z4l6" (UID: "09a07cf3-a768-4fcd-ade4-85783f4bf6d6") : secret "insights-runtime-extractor-tls" not found
Jul 30 02:30:15.615733 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:15.615649    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs\") pod \"network-metrics-daemon-wphj8\" (UID: \"ec397db7-4d16-4a1d-9ace-fb4de6e17039\") " pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:30:15.616084 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:15.615781    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 30 02:30:15.616084 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:15.615837    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs podName:ec397db7-4d16-4a1d-9ace-fb4de6e17039 nodeName:}" failed. No retries permitted until 2026-07-30 02:32:17.615822976 +0000 UTC m=+373.262606360 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs") pod "network-metrics-daemon-wphj8" (UID: "ec397db7-4d16-4a1d-9ace-fb4de6e17039") : secret "metrics-daemon-secret" not found
Jul 30 02:30:16.321390 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:16.321354    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:16.321586 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:16.321423    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:16.321586 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:16.321496    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 02:30:16.321586 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:16.321510    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 02:30:16.321586 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:16.321547    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-rbac-proxy-cm podName:09a07cf3-a768-4fcd-ade4-85783f4bf6d6 nodeName:}" failed. No retries permitted until 2026-07-30 02:30:18.321533961 +0000 UTC m=+253.968317347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9z4l6" (UID: "09a07cf3-a768-4fcd-ade4-85783f4bf6d6") : configmap "kube-rbac-proxy" not found
Jul 30 02:30:16.321586 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:16.321565    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-insights-runtime-extractor-tls podName:09a07cf3-a768-4fcd-ade4-85783f4bf6d6 nodeName:}" failed. No retries permitted until 2026-07-30 02:30:18.321552662 +0000 UTC m=+253.968336045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9z4l6" (UID: "09a07cf3-a768-4fcd-ade4-85783f4bf6d6") : secret "insights-runtime-extractor-tls" not found
Jul 30 02:30:16.615663 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:16.615591    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bzccb_648853ed-378d-4c9d-82e8-5cbb958399db/dns-node-resolver/0.log"
Jul 30 02:30:17.615415 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:17.615389    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bspvc_1f1a6683-1bcf-48ce-a20b-525984e9a23a/node-ca/0.log"
Jul 30 02:30:18.335763 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:18.335716    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:18.335958 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:18.335783    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:18.335958 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:18.335835    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 02:30:18.335958 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:18.335892    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 02:30:18.335958 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:18.335911    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-rbac-proxy-cm podName:09a07cf3-a768-4fcd-ade4-85783f4bf6d6 nodeName:}" failed. No retries permitted until 2026-07-30 02:30:22.335895031 +0000 UTC m=+257.982678414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9z4l6" (UID: "09a07cf3-a768-4fcd-ade4-85783f4bf6d6") : configmap "kube-rbac-proxy" not found
Jul 30 02:30:18.335958 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:18.335960    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-insights-runtime-extractor-tls podName:09a07cf3-a768-4fcd-ade4-85783f4bf6d6 nodeName:}" failed. No retries permitted until 2026-07-30 02:30:22.335943017 +0000 UTC m=+257.982726418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9z4l6" (UID: "09a07cf3-a768-4fcd-ade4-85783f4bf6d6") : secret "insights-runtime-extractor-tls" not found
Jul 30 02:30:18.615493 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:18.615423    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8gs9b_d4dfc729-47bc-41b7-b64d-0c1b45010108/serve-healthcheck-canary/0.log"
Jul 30 02:30:22.365702 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:22.365659    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:22.366052 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:22.365736    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:22.366052 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:22.365813    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 02:30:22.366052 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:22.365813    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 02:30:22.366052 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:22.365887    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-insights-runtime-extractor-tls podName:09a07cf3-a768-4fcd-ade4-85783f4bf6d6 nodeName:}" failed. No retries permitted until 2026-07-30 02:30:30.365872863 +0000 UTC m=+266.012656246 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9z4l6" (UID: "09a07cf3-a768-4fcd-ade4-85783f4bf6d6") : secret "insights-runtime-extractor-tls" not found
Jul 30 02:30:22.366052 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:22.365903    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-rbac-proxy-cm podName:09a07cf3-a768-4fcd-ade4-85783f4bf6d6 nodeName:}" failed. No retries permitted until 2026-07-30 02:30:30.365896328 +0000 UTC m=+266.012679711 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9z4l6" (UID: "09a07cf3-a768-4fcd-ade4-85783f4bf6d6") : configmap "kube-rbac-proxy" not found
Jul 30 02:30:30.431197 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:30.431093    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:30.431197 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:30.431155    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:30.431647 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:30.431629    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:30.433581 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:30.433556    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09a07cf3-a768-4fcd-ade4-85783f4bf6d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9z4l6\" (UID: \"09a07cf3-a768-4fcd-ade4-85783f4bf6d6\") " pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:30.526442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:30.526411    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9z4l6"
Jul 30 02:30:30.647711 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:30.647684    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9z4l6"]
Jul 30 02:30:30.650459 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:30:30.650426    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a07cf3_a768_4fcd_ade4_85783f4bf6d6.slice/crio-0cdf5ee55abe2fe4b85ecd2e68126a9a2838db2d5f613376fa6ea72cf4c0f67b WatchSource:0}: Error finding container 0cdf5ee55abe2fe4b85ecd2e68126a9a2838db2d5f613376fa6ea72cf4c0f67b: Status 404 returned error can't find the container with id 0cdf5ee55abe2fe4b85ecd2e68126a9a2838db2d5f613376fa6ea72cf4c0f67b
Jul 30 02:30:31.592200 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:31.592139    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9z4l6" event={"ID":"09a07cf3-a768-4fcd-ade4-85783f4bf6d6","Type":"ContainerStarted","Data":"0cdf5ee55abe2fe4b85ecd2e68126a9a2838db2d5f613376fa6ea72cf4c0f67b"}
Jul 30 02:30:34.861729 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:34.861690    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d95b6b5f4-ccppl"]
Jul 30 02:30:34.864808 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:34.864787    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:34.867706 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:34.867682    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 30 02:30:34.867706 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:34.867686    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 30 02:30:34.867906 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:34.867731    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6r6b9\""
Jul 30 02:30:34.867906 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:34.867751    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 30 02:30:34.874107 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:34.874084    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 30 02:30:34.879257 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:34.879237    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d95b6b5f4-ccppl"]
Jul 30 02:30:34.965037 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:34.965002    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-image-registry-private-configuration\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:34.965037 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:34.965048    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-installation-pull-secrets\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:34.965299 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:34.965074    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-bound-sa-token\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:34.965299 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:34.965155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-registry-tls\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:34.965299 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:34.965217    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdtvf\" (UniqueName: \"kubernetes.io/projected/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-kube-api-access-qdtvf\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:34.965299 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:34.965260    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-ca-trust-extracted\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:34.965475 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:34.965302    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-registry-certificates\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:34.965475 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:34.965335    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-trusted-ca\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:35.066479 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:35.066435    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-image-registry-private-configuration\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:35.066479 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:35.066477    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-installation-pull-secrets\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:35.066706 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:35.066494    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-bound-sa-token\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:35.066706 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:35.066532    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-registry-tls\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:35.066706 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:35.066562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdtvf\" (UniqueName: \"kubernetes.io/projected/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-kube-api-access-qdtvf\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:35.066706 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:35.066599    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-ca-trust-extracted\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:35.066706 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:35.066624    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-registry-certificates\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:35.066706 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:35.066658    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-trusted-ca\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:35.067162 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:35.067138    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-ca-trust-extracted\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:35.067897 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:35.067850    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-trusted-ca\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:35.067996 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:35.067928    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-registry-certificates\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:35.069569 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:35.069527    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-registry-tls\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:35.069679 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:35.069659    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-installation-pull-secrets\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:35.069724 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:35.069690    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-image-registry-private-configuration\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:35.087995 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:35.087968    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-bound-sa-token\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:35.088356 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:35.088336    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdtvf\" (UniqueName: \"kubernetes.io/projected/ad67c765-01ee-460c-b0ce-9da7d9c8a14e-kube-api-access-qdtvf\") pod \"image-registry-6d95b6b5f4-ccppl\" (UID: \"ad67c765-01ee-460c-b0ce-9da7d9c8a14e\") " pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:35.175522 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:35.175439    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:35.660296 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:35.660275    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d95b6b5f4-ccppl"]
Jul 30 02:30:35.662895 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:30:35.662870    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad67c765_01ee_460c_b0ce_9da7d9c8a14e.slice/crio-9cef70792999f405c390f4386d3d9f87792e9a380adfb3a56baf9a3472d13137 WatchSource:0}: Error finding container 9cef70792999f405c390f4386d3d9f87792e9a380adfb3a56baf9a3472d13137: Status 404 returned error can't find the container with id 9cef70792999f405c390f4386d3d9f87792e9a380adfb3a56baf9a3472d13137
Jul 30 02:30:36.609141 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:36.609102    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9z4l6" event={"ID":"09a07cf3-a768-4fcd-ade4-85783f4bf6d6","Type":"ContainerStarted","Data":"5a0c265b641b5a8af67926a084f312347249043c857971b9fe07d0ca0b96e147"}
Jul 30 02:30:36.610840 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:36.610780    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl" event={"ID":"ad67c765-01ee-460c-b0ce-9da7d9c8a14e","Type":"ContainerStarted","Data":"43694e51d048fed373cc7f9eedea4cdf3279a2c51e25ab522a63632d8dbefa96"}
Jul 30 02:30:36.610840 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:36.610826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl" event={"ID":"ad67c765-01ee-460c-b0ce-9da7d9c8a14e","Type":"ContainerStarted","Data":"9cef70792999f405c390f4386d3d9f87792e9a380adfb3a56baf9a3472d13137"}
Jul 30 02:30:36.611045 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:36.610890    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:39.623906 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:39.623870    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9z4l6" event={"ID":"09a07cf3-a768-4fcd-ade4-85783f4bf6d6","Type":"ContainerStarted","Data":"1804c85aaca7f6e86f2d51d3d5ec242fbfadb605de7c3b60007546bcd9222006"}
Jul 30 02:30:42.635397 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:42.635368    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9z4l6" event={"ID":"09a07cf3-a768-4fcd-ade4-85783f4bf6d6","Type":"ContainerStarted","Data":"756f4908029e8d118d58728a54fda210665c0eed6459b56648f067fa146f44fd"}
Jul 30 02:30:42.652510 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:42.652468    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9z4l6" podStartSLOduration=16.89950571 podStartE2EDuration="28.652428452s" totalImagesPullingTime="11.752922742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 02:30:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:30:42.651083286 +0000 UTC m=+278.297866691" watchObservedRunningTime="2026-07-30 02:30:42.652428452 +0000 UTC m=+278.299211852"
Jul 30 02:30:42.653243 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:42.653214    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl" podStartSLOduration=8.653202073 podStartE2EDuration="8.653202073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:30:36.631104498 +0000 UTC m=+272.277887904" watchObservedRunningTime="2026-07-30 02:30:42.653202073 +0000 UTC m=+278.299985478"
Jul 30 02:30:45.307151 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:45.307119    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-kdwrm"]
Jul 30 02:30:45.309997 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:45.309981    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-kdwrm"
Jul 30 02:30:45.312724 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:45.312702    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 30 02:30:45.312828 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:45.312780    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-8m6pr\""
Jul 30 02:30:45.317882 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:45.317860    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-kdwrm"]
Jul 30 02:30:45.345463 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:45.345434    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/65ed1700-1134-46d6-85f1-2619f1d9986f-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-kdwrm\" (UID: \"65ed1700-1134-46d6-85f1-2619f1d9986f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-kdwrm"
Jul 30 02:30:45.446388 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:45.446350    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/65ed1700-1134-46d6-85f1-2619f1d9986f-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-kdwrm\" (UID: \"65ed1700-1134-46d6-85f1-2619f1d9986f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-kdwrm"
Jul 30 02:30:45.448822 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:45.448799    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/65ed1700-1134-46d6-85f1-2619f1d9986f-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-kdwrm\" (UID: \"65ed1700-1134-46d6-85f1-2619f1d9986f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-kdwrm"
Jul 30 02:30:45.619472 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:45.619392    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-kdwrm"
Jul 30 02:30:45.742093 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:45.742072    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-kdwrm"]
Jul 30 02:30:45.744503 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:30:45.744477    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ed1700_1134_46d6_85f1_2619f1d9986f.slice/crio-a3f6c390e5cc6706bf0c0c06369f2ea736d8fdd60c2d4810568d8414769de7fd WatchSource:0}: Error finding container a3f6c390e5cc6706bf0c0c06369f2ea736d8fdd60c2d4810568d8414769de7fd: Status 404 returned error can't find the container with id a3f6c390e5cc6706bf0c0c06369f2ea736d8fdd60c2d4810568d8414769de7fd
Jul 30 02:30:46.648868 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:46.648828    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-kdwrm" event={"ID":"65ed1700-1134-46d6-85f1-2619f1d9986f","Type":"ContainerStarted","Data":"a3f6c390e5cc6706bf0c0c06369f2ea736d8fdd60c2d4810568d8414769de7fd"}
Jul 30 02:30:47.298295 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:47.298257    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7rhjt" podUID="6641518e-6c07-41e1-92b0-470be0e94ddc"
Jul 30 02:30:47.654181 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:47.654075    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:30:47.654181 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:47.654101    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-kdwrm" event={"ID":"65ed1700-1134-46d6-85f1-2619f1d9986f","Type":"ContainerStarted","Data":"bd546720e17d72b87df3195be04909e22b45a84e636e79ab94b64de14672c3f4"}
Jul 30 02:30:47.654607 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:47.654439    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-kdwrm"
Jul 30 02:30:47.658903 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:47.658885    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-kdwrm"
Jul 30 02:30:47.672912 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:47.672825    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-kdwrm" podStartSLOduration=1.518265296 podStartE2EDuration="2.672813429s" totalImagesPullingTime="1.154548133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:30:47.672363617 +0000 UTC m=+283.319147021" watchObservedRunningTime="2026-07-30 02:30:47.672813429 +0000 UTC m=+283.319596834"
Jul 30 02:30:50.681630 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:50.681595    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:30:50.684037 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:50.684019    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6641518e-6c07-41e1-92b0-470be0e94ddc-metrics-tls\") pod \"dns-default-7rhjt\" (UID: \"6641518e-6c07-41e1-92b0-470be0e94ddc\") " pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:30:50.957677 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:50.957584    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5sqnz\""
Jul 30 02:30:50.964949 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:50.964925    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:30:51.089030 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:51.089001    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7rhjt"]
Jul 30 02:30:51.091916 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:30:51.091890    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6641518e_6c07_41e1_92b0_470be0e94ddc.slice/crio-13e9d411ea2fc24b55d2f1a147a011b99f4fb23c02de5a9fc1069daebe23ea75 WatchSource:0}: Error finding container 13e9d411ea2fc24b55d2f1a147a011b99f4fb23c02de5a9fc1069daebe23ea75: Status 404 returned error can't find the container with id 13e9d411ea2fc24b55d2f1a147a011b99f4fb23c02de5a9fc1069daebe23ea75
Jul 30 02:30:51.667426 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:51.667383    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7rhjt" event={"ID":"6641518e-6c07-41e1-92b0-470be0e94ddc","Type":"ContainerStarted","Data":"13e9d411ea2fc24b55d2f1a147a011b99f4fb23c02de5a9fc1069daebe23ea75"}
Jul 30 02:30:53.674623 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.674584    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7rhjt" event={"ID":"6641518e-6c07-41e1-92b0-470be0e94ddc","Type":"ContainerStarted","Data":"9f46193a11fac5394c72b25592085123150f47be102892112e5e486dcd209b8f"}
Jul 30 02:30:53.675095 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.674644    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7rhjt" event={"ID":"6641518e-6c07-41e1-92b0-470be0e94ddc","Type":"ContainerStarted","Data":"80b6eff443ea318a96e9b552dc713130a93dca2f53cba04462ae4053701de13e"}
Jul 30 02:30:53.675095 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.674729    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:30:53.687732 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.687703    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dn7g6"]
Jul 30 02:30:53.690622 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.690607    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.693542 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.693514    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 30 02:30:53.693542 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.693514    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 30 02:30:53.693703 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.693570    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 30 02:30:53.693703 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.693570    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 30 02:30:53.695205 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.695191    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h2qjz\""
Jul 30 02:30:53.695205 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.695201    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 30 02:30:53.695337 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.695225    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 30 02:30:53.700569 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.700516    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7rhjt" podStartSLOduration=252.109359398 podStartE2EDuration="4m13.700503684s" totalImagesPullingTime="1.591144286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:30:53.699808782 +0000 UTC m=+289.346592187" watchObservedRunningTime="2026-07-30 02:30:53.700503684 +0000 UTC m=+289.347287088"
Jul 30 02:30:53.702409 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.702391    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-root\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.702510 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.702416    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-node-exporter-tls\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.702510 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.702437    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-node-exporter-textfile\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.702510 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.702451    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-node-exporter-wtmp\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.702510 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.702474    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.702671 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.702529    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxvgd\" (UniqueName: \"kubernetes.io/projected/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-kube-api-access-nxvgd\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.702671 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.702569    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-metrics-client-ca\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.702671 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.702591    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-sys\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.702671 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.702614    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-node-exporter-accelerators-collector-config\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.708766 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.708745    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"]
Jul 30 02:30:53.711665 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.711650    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"
Jul 30 02:30:53.714301 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.714282    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 30 02:30:53.714436 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.714401    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4mxb6\""
Jul 30 02:30:53.714573 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.714543    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 30 02:30:53.714822 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.714805    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 30 02:30:53.722008 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.721932    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"]
Jul 30 02:30:53.803702 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.803673    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-node-exporter-textfile\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.803702 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.803703    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-node-exporter-wtmp\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.803940 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.803720    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.803940 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.803750    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ced05f60-22f9-4ae7-a971-d530833ac4ad-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-xn726\" (UID: \"ced05f60-22f9-4ae7-a971-d530833ac4ad\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"
Jul 30 02:30:53.803940 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.803823    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxvgd\" (UniqueName: \"kubernetes.io/projected/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-kube-api-access-nxvgd\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.803940 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.803865    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-node-exporter-wtmp\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.803940 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.803871    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ced05f60-22f9-4ae7-a971-d530833ac4ad-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-xn726\" (UID: \"ced05f60-22f9-4ae7-a971-d530833ac4ad\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"
Jul 30 02:30:53.803940 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.803931    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-metrics-client-ca\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.804272 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.803956    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-sys\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.804272 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.803988    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-node-exporter-accelerators-collector-config\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.804272 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.804015    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf7bg\" (UniqueName: \"kubernetes.io/projected/ced05f60-22f9-4ae7-a971-d530833ac4ad-kube-api-access-mf7bg\") pod \"kube-state-metrics-56d9f944b4-xn726\" (UID: \"ced05f60-22f9-4ae7-a971-d530833ac4ad\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"
Jul 30 02:30:53.804272 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.804044    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-node-exporter-textfile\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.804272 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.804052    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-sys\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.804272 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.804113    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ced05f60-22f9-4ae7-a971-d530833ac4ad-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-xn726\" (UID: \"ced05f60-22f9-4ae7-a971-d530833ac4ad\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"
Jul 30 02:30:53.804272 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.804151    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-root\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.804272 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.804196    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ced05f60-22f9-4ae7-a971-d530833ac4ad-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-xn726\" (UID: \"ced05f60-22f9-4ae7-a971-d530833ac4ad\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"
Jul 30 02:30:53.804272 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.804222    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-root\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.804272 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.804225    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-node-exporter-tls\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.804272 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.804260    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ced05f60-22f9-4ae7-a971-d530833ac4ad-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-xn726\" (UID: \"ced05f60-22f9-4ae7-a971-d530833ac4ad\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"
Jul 30 02:30:53.804731 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:53.804284    2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 30 02:30:53.804731 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:53.804335    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-node-exporter-tls podName:4ce518f3-8390-4e94-b88a-cdbe86cc1d7b nodeName:}" failed. No retries permitted until 2026-07-30 02:30:54.304320991 +0000 UTC m=+289.951104374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-node-exporter-tls") pod "node-exporter-dn7g6" (UID: "4ce518f3-8390-4e94-b88a-cdbe86cc1d7b") : secret "node-exporter-tls" not found
Jul 30 02:30:53.804731 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.804559    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-metrics-client-ca\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.804828 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.804739    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-node-exporter-accelerators-collector-config\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.806225 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.806207    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.816247 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.816222    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxvgd\" (UniqueName: \"kubernetes.io/projected/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-kube-api-access-nxvgd\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:53.905615 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.905577    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ced05f60-22f9-4ae7-a971-d530833ac4ad-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-xn726\" (UID: \"ced05f60-22f9-4ae7-a971-d530833ac4ad\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"
Jul 30 02:30:53.905615 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.905618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ced05f60-22f9-4ae7-a971-d530833ac4ad-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-xn726\" (UID: \"ced05f60-22f9-4ae7-a971-d530833ac4ad\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"
Jul 30 02:30:53.905854 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.905652    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ced05f60-22f9-4ae7-a971-d530833ac4ad-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-xn726\" (UID: \"ced05f60-22f9-4ae7-a971-d530833ac4ad\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"
Jul 30 02:30:53.905854 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.905671    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ced05f60-22f9-4ae7-a971-d530833ac4ad-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-xn726\" (UID: \"ced05f60-22f9-4ae7-a971-d530833ac4ad\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"
Jul 30 02:30:53.905854 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.905691    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ced05f60-22f9-4ae7-a971-d530833ac4ad-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-xn726\" (UID: \"ced05f60-22f9-4ae7-a971-d530833ac4ad\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"
Jul 30 02:30:53.905854 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.905727    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf7bg\" (UniqueName: \"kubernetes.io/projected/ced05f60-22f9-4ae7-a971-d530833ac4ad-kube-api-access-mf7bg\") pod \"kube-state-metrics-56d9f944b4-xn726\" (UID: \"ced05f60-22f9-4ae7-a971-d530833ac4ad\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"
Jul 30 02:30:53.906124 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.906087    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ced05f60-22f9-4ae7-a971-d530833ac4ad-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-xn726\" (UID: \"ced05f60-22f9-4ae7-a971-d530833ac4ad\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"
Jul 30 02:30:53.906453 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.906432    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ced05f60-22f9-4ae7-a971-d530833ac4ad-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-xn726\" (UID: \"ced05f60-22f9-4ae7-a971-d530833ac4ad\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"
Jul 30 02:30:53.906707 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.906681    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ced05f60-22f9-4ae7-a971-d530833ac4ad-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-xn726\" (UID: \"ced05f60-22f9-4ae7-a971-d530833ac4ad\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"
Jul 30 02:30:53.908292 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.908269    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ced05f60-22f9-4ae7-a971-d530833ac4ad-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-xn726\" (UID: \"ced05f60-22f9-4ae7-a971-d530833ac4ad\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"
Jul 30 02:30:53.908341 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.908283    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ced05f60-22f9-4ae7-a971-d530833ac4ad-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-xn726\" (UID: \"ced05f60-22f9-4ae7-a971-d530833ac4ad\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"
Jul 30 02:30:53.913140 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:53.913115    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf7bg\" (UniqueName: \"kubernetes.io/projected/ced05f60-22f9-4ae7-a971-d530833ac4ad-kube-api-access-mf7bg\") pod \"kube-state-metrics-56d9f944b4-xn726\" (UID: \"ced05f60-22f9-4ae7-a971-d530833ac4ad\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"
Jul 30 02:30:54.020951 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.020844    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"
Jul 30 02:30:54.151393 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.151320    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-xn726"]
Jul 30 02:30:54.153722 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:30:54.153695    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced05f60_22f9_4ae7_a971_d530833ac4ad.slice/crio-4a566b6af3a682180306713b7a3e21136b6f9364296f8e6fa42d1e06c260911c WatchSource:0}: Error finding container 4a566b6af3a682180306713b7a3e21136b6f9364296f8e6fa42d1e06c260911c: Status 404 returned error can't find the container with id 4a566b6af3a682180306713b7a3e21136b6f9364296f8e6fa42d1e06c260911c
Jul 30 02:30:54.310060 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.310022    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-node-exporter-tls\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:54.312423 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.312396    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ce518f3-8390-4e94-b88a-cdbe86cc1d7b-node-exporter-tls\") pod \"node-exporter-dn7g6\" (UID: \"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b\") " pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:54.598975 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.598884    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dn7g6"
Jul 30 02:30:54.609978 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:30:54.609948    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce518f3_8390_4e94_b88a_cdbe86cc1d7b.slice/crio-5e3abae2a013440e9bf32c257c241866a0c976bd06b65e1d105696ab6c328c5c WatchSource:0}: Error finding container 5e3abae2a013440e9bf32c257c241866a0c976bd06b65e1d105696ab6c328c5c: Status 404 returned error can't find the container with id 5e3abae2a013440e9bf32c257c241866a0c976bd06b65e1d105696ab6c328c5c
Jul 30 02:30:54.678933 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.678890    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dn7g6" event={"ID":"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b","Type":"ContainerStarted","Data":"5e3abae2a013440e9bf32c257c241866a0c976bd06b65e1d105696ab6c328c5c"}
Jul 30 02:30:54.679946 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.679923    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726" event={"ID":"ced05f60-22f9-4ae7-a971-d530833ac4ad","Type":"ContainerStarted","Data":"4a566b6af3a682180306713b7a3e21136b6f9364296f8e6fa42d1e06c260911c"}
Jul 30 02:30:54.756948 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.756916    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 02:30:54.761348 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.761332    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.764247 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.764225    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 30 02:30:54.764382 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.764308    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 30 02:30:54.764382 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.764324    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 30 02:30:54.764382 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.764356    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 30 02:30:54.764534 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.764358    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 30 02:30:54.764534 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.764443    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 30 02:30:54.764676 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.764638    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 30 02:30:54.764676 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.764643    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 30 02:30:54.764851 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.764691    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 30 02:30:54.764851 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.764741    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xfjt2\""
Jul 30 02:30:54.775111 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.775073    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 02:30:54.813649 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.813615    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.813798 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.813667    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-config-volume\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.813798 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.813701    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41df2d2c-e95d-488a-b18f-8862a6239cb3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.813798 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.813728    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41df2d2c-e95d-488a-b18f-8862a6239cb3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.813798 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.813754    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.813798 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.813779    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41df2d2c-e95d-488a-b18f-8862a6239cb3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.814076 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.813812    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.814076 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.813843    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41df2d2c-e95d-488a-b18f-8862a6239cb3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.814076 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.813924    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ln92\" (UniqueName: \"kubernetes.io/projected/41df2d2c-e95d-488a-b18f-8862a6239cb3-kube-api-access-7ln92\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.814076 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.813966    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.814076 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.814025    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41df2d2c-e95d-488a-b18f-8862a6239cb3-config-out\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.814076 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.814064    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-web-config\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.814295 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.814145    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.915029 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.914954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.915029 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.914994    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41df2d2c-e95d-488a-b18f-8862a6239cb3-config-out\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.915029 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.915019    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-web-config\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.915319 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.915199    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.915319 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.915266    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.915319 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.915303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-config-volume\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.915465 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.915338    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41df2d2c-e95d-488a-b18f-8862a6239cb3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.915465 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.915369    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41df2d2c-e95d-488a-b18f-8862a6239cb3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.915465 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.915394    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.915465 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.915419    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41df2d2c-e95d-488a-b18f-8862a6239cb3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.915465 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.915457    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.915695 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.915493    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41df2d2c-e95d-488a-b18f-8862a6239cb3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.915695 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.915524    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ln92\" (UniqueName: \"kubernetes.io/projected/41df2d2c-e95d-488a-b18f-8862a6239cb3-kube-api-access-7ln92\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.917575 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.916273    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41df2d2c-e95d-488a-b18f-8862a6239cb3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.917575 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:54.916387    2565 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 30 02:30:54.917575 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:30:54.916606    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-main-tls podName:41df2d2c-e95d-488a-b18f-8862a6239cb3 nodeName:}" failed. No retries permitted until 2026-07-30 02:30:55.416586749 +0000 UTC m=+291.063370133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "41df2d2c-e95d-488a-b18f-8862a6239cb3") : secret "alertmanager-main-tls" not found
Jul 30 02:30:54.917575 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.916979    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41df2d2c-e95d-488a-b18f-8862a6239cb3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.917575 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.917469    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41df2d2c-e95d-488a-b18f-8862a6239cb3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.918624 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.918339    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.918624 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.918473    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.918793 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.918647    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41df2d2c-e95d-488a-b18f-8862a6239cb3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.918793 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.918690    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.918793 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.918707    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-config-volume\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.919093 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.919071    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-web-config\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.919153 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.919119    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41df2d2c-e95d-488a-b18f-8862a6239cb3-config-out\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.919841 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.919823    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:54.923478 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:54.923459    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ln92\" (UniqueName: \"kubernetes.io/projected/41df2d2c-e95d-488a-b18f-8862a6239cb3-kube-api-access-7ln92\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:55.180226 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:55.180118    2565 patch_prober.go:28] interesting pod/image-registry-6d95b6b5f4-ccppl container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 30 02:30:55.180226 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:55.180203    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl" podUID="ad67c765-01ee-460c-b0ce-9da7d9c8a14e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 02:30:55.420078 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:55.420049    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:55.422439 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:55.422415    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:55.673247 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:55.673219    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:30:55.685215 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:55.685157    2565 generic.go:356] "Generic (PLEG): container finished" podID="4ce518f3-8390-4e94-b88a-cdbe86cc1d7b" containerID="1a5709d75508f736170238b070380b8e83d69cb6a9c64ade4937cd9f0cfea255" exitCode=0
Jul 30 02:30:55.685565 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:55.685242    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dn7g6" event={"ID":"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b","Type":"ContainerDied","Data":"1a5709d75508f736170238b070380b8e83d69cb6a9c64ade4937cd9f0cfea255"}
Jul 30 02:30:55.841835 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:55.841810    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 02:30:55.846415 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:30:55.846356    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41df2d2c_e95d_488a_b18f_8862a6239cb3.slice/crio-532acc3ab5a80f2de83803de515572535fbae91468eb7b01be1072a16187a98c WatchSource:0}: Error finding container 532acc3ab5a80f2de83803de515572535fbae91468eb7b01be1072a16187a98c: Status 404 returned error can't find the container with id 532acc3ab5a80f2de83803de515572535fbae91468eb7b01be1072a16187a98c
Jul 30 02:30:55.853377 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:55.853351    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-tjx92"]
Jul 30 02:30:55.857659 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:55.857641    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-tjx92"
Jul 30 02:30:55.860387 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:55.860332    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 30 02:30:55.860484 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:55.860408    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-h55kn\""
Jul 30 02:30:55.860538 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:55.860513    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 30 02:30:55.865418 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:55.865043    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-tjx92"]
Jul 30 02:30:55.925240 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:55.925200    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhs82\" (UniqueName: \"kubernetes.io/projected/e16dc4ef-22ab-442d-b030-b5903e9c92f1-kube-api-access-qhs82\") pod \"downloads-7c97dc4555-tjx92\" (UID: \"e16dc4ef-22ab-442d-b030-b5903e9c92f1\") " pod="openshift-console/downloads-7c97dc4555-tjx92"
Jul 30 02:30:56.026279 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:56.026189    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhs82\" (UniqueName: \"kubernetes.io/projected/e16dc4ef-22ab-442d-b030-b5903e9c92f1-kube-api-access-qhs82\") pod \"downloads-7c97dc4555-tjx92\" (UID: \"e16dc4ef-22ab-442d-b030-b5903e9c92f1\") " pod="openshift-console/downloads-7c97dc4555-tjx92"
Jul 30 02:30:56.035181 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:56.035132    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhs82\" (UniqueName: \"kubernetes.io/projected/e16dc4ef-22ab-442d-b030-b5903e9c92f1-kube-api-access-qhs82\") pod \"downloads-7c97dc4555-tjx92\" (UID: \"e16dc4ef-22ab-442d-b030-b5903e9c92f1\") " pod="openshift-console/downloads-7c97dc4555-tjx92"
Jul 30 02:30:56.170237 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:56.170200    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-tjx92"
Jul 30 02:30:56.352327 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:30:56.352297    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16dc4ef_22ab_442d_b030_b5903e9c92f1.slice/crio-fa1928a1022a1caeb5730677dbc52a45e8b5cffc170cd046ac04666e86c268d0 WatchSource:0}: Error finding container fa1928a1022a1caeb5730677dbc52a45e8b5cffc170cd046ac04666e86c268d0: Status 404 returned error can't find the container with id fa1928a1022a1caeb5730677dbc52a45e8b5cffc170cd046ac04666e86c268d0
Jul 30 02:30:56.352327 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:56.352305    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-tjx92"]
Jul 30 02:30:56.690497 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:56.690454    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41df2d2c-e95d-488a-b18f-8862a6239cb3","Type":"ContainerStarted","Data":"532acc3ab5a80f2de83803de515572535fbae91468eb7b01be1072a16187a98c"}
Jul 30 02:30:56.693076 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:56.693047    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dn7g6" event={"ID":"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b","Type":"ContainerStarted","Data":"a9d03b5e8cebfe209cfd43b54491655d471a7ff11835a6a39b9cf26214450465"}
Jul 30 02:30:56.693235 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:56.693106    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dn7g6" event={"ID":"4ce518f3-8390-4e94-b88a-cdbe86cc1d7b","Type":"ContainerStarted","Data":"1db787b569cdf08db1f35a44c2301125a659843aef44767146fbf3d7228f2127"}
Jul 30 02:30:56.695541 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:56.695520    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726" event={"ID":"ced05f60-22f9-4ae7-a971-d530833ac4ad","Type":"ContainerStarted","Data":"fcf9e04b4ac443019d421fc0191d11a417b8b73cac3aeb2a21b2cb244b75e5da"}
Jul 30 02:30:56.695629 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:56.695580    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726" event={"ID":"ced05f60-22f9-4ae7-a971-d530833ac4ad","Type":"ContainerStarted","Data":"c513cfe7b52154c14ca095dc85c854e6c8c19d6d781339e1ab06d3e6a9fe71da"}
Jul 30 02:30:56.695629 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:56.695611    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726" event={"ID":"ced05f60-22f9-4ae7-a971-d530833ac4ad","Type":"ContainerStarted","Data":"3171b3b905f4b161b6f84bf4c8f0d617de350c8f04c5c9baa3b08a877f2af8ee"}
Jul 30 02:30:56.696990 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:56.696969    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-tjx92" event={"ID":"e16dc4ef-22ab-442d-b030-b5903e9c92f1","Type":"ContainerStarted","Data":"fa1928a1022a1caeb5730677dbc52a45e8b5cffc170cd046ac04666e86c268d0"}
Jul 30 02:30:56.714841 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:56.714751    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dn7g6" podStartSLOduration=2.841247063 podStartE2EDuration="3.71473792s" totalImagesPullingTime="794.869517ms" totalInitContainerRuntime="78.62134ms" isStatefulPod=true podCreationTimestamp="2026-07-30 02:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:30:56.71251771 +0000 UTC m=+292.359301134" watchObservedRunningTime="2026-07-30 02:30:56.71473792 +0000 UTC m=+292.361521325"
Jul 30 02:30:56.732879 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:56.732834    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xn726" podStartSLOduration=1.639710671 podStartE2EDuration="3.732819869s" totalImagesPullingTime="2.093109198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:30:56.730574446 +0000 UTC m=+292.377357863" watchObservedRunningTime="2026-07-30 02:30:56.732819869 +0000 UTC m=+292.379603273"
Jul 30 02:30:57.622195 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:57.622139    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d95b6b5f4-ccppl"
Jul 30 02:30:57.704264 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:57.704204    2565 generic.go:356] "Generic (PLEG): container finished" podID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerID="b12302717ee2ad8a1b5144ac8cd3c5336dacaa864c6f0f5d10bf377f76e239c3" exitCode=0
Jul 30 02:30:57.705889 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:57.705862    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41df2d2c-e95d-488a-b18f-8862a6239cb3","Type":"ContainerDied","Data":"b12302717ee2ad8a1b5144ac8cd3c5336dacaa864c6f0f5d10bf377f76e239c3"}
Jul 30 02:30:59.716983 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:59.716956    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41df2d2c-e95d-488a-b18f-8862a6239cb3","Type":"ContainerStarted","Data":"1cb14375915dea2a045ec18dbec5eec4c1184479aa49d70ed262b9f338172ad8"}
Jul 30 02:30:59.717301 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:30:59.717023    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41df2d2c-e95d-488a-b18f-8862a6239cb3","Type":"ContainerStarted","Data":"9850e8c41f599bfb44c91bc12ee4a909bc73cdf2963956efa3d329b27a7d9e8f"}
Jul 30 02:31:00.729956 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:00.729919    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41df2d2c-e95d-488a-b18f-8862a6239cb3","Type":"ContainerStarted","Data":"2df0bc1eeffa2804c75b4950b4f788a98491b117ffbb4a07f144f68fcb30dcfb"}
Jul 30 02:31:00.730452 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:00.730019    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41df2d2c-e95d-488a-b18f-8862a6239cb3","Type":"ContainerStarted","Data":"60d862d2860b781a83f2d351359146e1db3068837508caf826b22dac72f0579a"}
Jul 30 02:31:00.730452 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:00.730055    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41df2d2c-e95d-488a-b18f-8862a6239cb3","Type":"ContainerStarted","Data":"698a6fd58c238d2c7c326892367d49a135bb89338c592e94b51dac98f95ded40"}
Jul 30 02:31:01.738040 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:01.737995    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41df2d2c-e95d-488a-b18f-8862a6239cb3","Type":"ContainerStarted","Data":"7001017cf0eed65701f29f947559f35be94ff1f344625e4702091fe8739fd535"}
Jul 30 02:31:01.766915 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:01.766865    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.698254212 podStartE2EDuration="7.766848872s" totalImagesPullingTime="4.054087298s" totalInitContainerRuntime="14.507362ms" isStatefulPod=false podCreationTimestamp="2026-07-30 02:30:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:31:01.764724164 +0000 UTC m=+297.411507582" watchObservedRunningTime="2026-07-30 02:31:01.766848872 +0000 UTC m=+297.413632298"
Jul 30 02:31:01.796653 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:31:01.796615    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-wphj8" podUID="ec397db7-4d16-4a1d-9ace-fb4de6e17039"
Jul 30 02:31:03.682491 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:03.682458    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7rhjt"
Jul 30 02:31:04.696424 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:04.696397    2565 kubelet.go:1648] "Image garbage collection succeeded"
Jul 30 02:31:13.786823 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:13.786741    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-tjx92" event={"ID":"e16dc4ef-22ab-442d-b030-b5903e9c92f1","Type":"ContainerStarted","Data":"81e03e407e3c2b15f529769991d6e00faf71cbd02a633e411041d115d5234ed8"}
Jul 30 02:31:13.787315 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:13.786944    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-tjx92"
Jul 30 02:31:13.800688 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:13.800659    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-tjx92"
Jul 30 02:31:13.806447 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:13.806398    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-tjx92" podStartSLOduration=1.7066765830000001 podStartE2EDuration="18.806381992s" totalImagesPullingTime="17.099705409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:31:13.805548176 +0000 UTC m=+309.452331600" watchObservedRunningTime="2026-07-30 02:31:13.806381992 +0000 UTC m=+309.453165400"
Jul 30 02:31:14.796743 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:14.796709    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:31:25.808869 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:25.808831    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7" podUID="af4e9838-5365-4fea-aed4-3200eacf14c4" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 30 02:31:35.808973 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:35.808935    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7" podUID="af4e9838-5365-4fea-aed4-3200eacf14c4" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 30 02:31:45.808296 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:45.808253    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7" podUID="af4e9838-5365-4fea-aed4-3200eacf14c4" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 30 02:31:45.808670 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:45.808317    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7"
Jul 30 02:31:45.808776 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:45.808746    2565 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"3bc1d74bd89273cadefc74fb15ca2cb6b380409967bfcab0f9a9f37eaa0cf83a"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 30 02:31:45.808813 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:45.808798    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7" podUID="af4e9838-5365-4fea-aed4-3200eacf14c4" containerName="service-proxy" containerID="cri-o://3bc1d74bd89273cadefc74fb15ca2cb6b380409967bfcab0f9a9f37eaa0cf83a" gracePeriod=30
Jul 30 02:31:46.903257 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:46.903226    2565 generic.go:356] "Generic (PLEG): container finished" podID="af4e9838-5365-4fea-aed4-3200eacf14c4" containerID="3bc1d74bd89273cadefc74fb15ca2cb6b380409967bfcab0f9a9f37eaa0cf83a" exitCode=2
Jul 30 02:31:46.903815 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:46.903303    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7" event={"ID":"af4e9838-5365-4fea-aed4-3200eacf14c4","Type":"ContainerDied","Data":"3bc1d74bd89273cadefc74fb15ca2cb6b380409967bfcab0f9a9f37eaa0cf83a"}
Jul 30 02:31:46.903815 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:31:46.903362    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76d97b5f99-hgzj7" event={"ID":"af4e9838-5365-4fea-aed4-3200eacf14c4","Type":"ContainerStarted","Data":"62c98cfed5e4208b43ed1cda8785ba09e74852d6536bf51cd3001303d19e0054"}
Jul 30 02:32:14.023984 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:14.023902    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 02:32:14.024445 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:14.024200    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="alertmanager" containerID="cri-o://9850e8c41f599bfb44c91bc12ee4a909bc73cdf2963956efa3d329b27a7d9e8f" gracePeriod=120
Jul 30 02:32:14.024445 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:14.024248    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="kube-rbac-proxy-metric" containerID="cri-o://2df0bc1eeffa2804c75b4950b4f788a98491b117ffbb4a07f144f68fcb30dcfb" gracePeriod=120
Jul 30 02:32:14.024445 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:14.024287    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="kube-rbac-proxy-web" containerID="cri-o://698a6fd58c238d2c7c326892367d49a135bb89338c592e94b51dac98f95ded40" gracePeriod=120
Jul 30 02:32:14.024445 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:14.024313    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="prom-label-proxy" containerID="cri-o://7001017cf0eed65701f29f947559f35be94ff1f344625e4702091fe8739fd535" gracePeriod=120
Jul 30 02:32:14.024445 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:14.024386    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="kube-rbac-proxy" containerID="cri-o://60d862d2860b781a83f2d351359146e1db3068837508caf826b22dac72f0579a" gracePeriod=120
Jul 30 02:32:14.024703 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:14.024345    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="config-reloader" containerID="cri-o://1cb14375915dea2a045ec18dbec5eec4c1184479aa49d70ed262b9f338172ad8" gracePeriod=120
Jul 30 02:32:15.004304 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.004271    2565 generic.go:356] "Generic (PLEG): container finished" podID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerID="7001017cf0eed65701f29f947559f35be94ff1f344625e4702091fe8739fd535" exitCode=0
Jul 30 02:32:15.004304 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.004297    2565 generic.go:356] "Generic (PLEG): container finished" podID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerID="60d862d2860b781a83f2d351359146e1db3068837508caf826b22dac72f0579a" exitCode=0
Jul 30 02:32:15.004304 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.004305    2565 generic.go:356] "Generic (PLEG): container finished" podID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerID="698a6fd58c238d2c7c326892367d49a135bb89338c592e94b51dac98f95ded40" exitCode=0
Jul 30 02:32:15.004304 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.004311    2565 generic.go:356] "Generic (PLEG): container finished" podID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerID="1cb14375915dea2a045ec18dbec5eec4c1184479aa49d70ed262b9f338172ad8" exitCode=0
Jul 30 02:32:15.004304 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.004315    2565 generic.go:356] "Generic (PLEG): container finished" podID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerID="9850e8c41f599bfb44c91bc12ee4a909bc73cdf2963956efa3d329b27a7d9e8f" exitCode=0
Jul 30 02:32:15.004587 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.004346    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41df2d2c-e95d-488a-b18f-8862a6239cb3","Type":"ContainerDied","Data":"7001017cf0eed65701f29f947559f35be94ff1f344625e4702091fe8739fd535"}
Jul 30 02:32:15.004587 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.004419    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41df2d2c-e95d-488a-b18f-8862a6239cb3","Type":"ContainerDied","Data":"60d862d2860b781a83f2d351359146e1db3068837508caf826b22dac72f0579a"}
Jul 30 02:32:15.004587 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.004450    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41df2d2c-e95d-488a-b18f-8862a6239cb3","Type":"ContainerDied","Data":"698a6fd58c238d2c7c326892367d49a135bb89338c592e94b51dac98f95ded40"}
Jul 30 02:32:15.004587 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.004481    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41df2d2c-e95d-488a-b18f-8862a6239cb3","Type":"ContainerDied","Data":"1cb14375915dea2a045ec18dbec5eec4c1184479aa49d70ed262b9f338172ad8"}
Jul 30 02:32:15.004587 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.004509    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41df2d2c-e95d-488a-b18f-8862a6239cb3","Type":"ContainerDied","Data":"9850e8c41f599bfb44c91bc12ee4a909bc73cdf2963956efa3d329b27a7d9e8f"}
Jul 30 02:32:15.272430 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.272406    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:15.398637 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.398602    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41df2d2c-e95d-488a-b18f-8862a6239cb3-tls-assets\") pod \"41df2d2c-e95d-488a-b18f-8862a6239cb3\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") "
Jul 30 02:32:15.398637 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.398640    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-web-config\") pod \"41df2d2c-e95d-488a-b18f-8862a6239cb3\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") "
Jul 30 02:32:15.398862 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.398661    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-cluster-tls-config\") pod \"41df2d2c-e95d-488a-b18f-8862a6239cb3\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") "
Jul 30 02:32:15.398862 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.398682    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-kube-rbac-proxy\") pod \"41df2d2c-e95d-488a-b18f-8862a6239cb3\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") "
Jul 30 02:32:15.398862 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.398702    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ln92\" (UniqueName: \"kubernetes.io/projected/41df2d2c-e95d-488a-b18f-8862a6239cb3-kube-api-access-7ln92\") pod \"41df2d2c-e95d-488a-b18f-8862a6239cb3\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") "
Jul 30 02:32:15.398862 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.398803    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41df2d2c-e95d-488a-b18f-8862a6239cb3-alertmanager-main-db\") pod \"41df2d2c-e95d-488a-b18f-8862a6239cb3\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") "
Jul 30 02:32:15.398862 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.398833    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41df2d2c-e95d-488a-b18f-8862a6239cb3-metrics-client-ca\") pod \"41df2d2c-e95d-488a-b18f-8862a6239cb3\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") "
Jul 30 02:32:15.398862 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.398857    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-kube-rbac-proxy-web\") pod \"41df2d2c-e95d-488a-b18f-8862a6239cb3\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") "
Jul 30 02:32:15.399204 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.398908    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41df2d2c-e95d-488a-b18f-8862a6239cb3-alertmanager-trusted-ca-bundle\") pod \"41df2d2c-e95d-488a-b18f-8862a6239cb3\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") "
Jul 30 02:32:15.399204 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.398937    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"41df2d2c-e95d-488a-b18f-8862a6239cb3\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") "
Jul 30 02:32:15.399204 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.398993    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-main-tls\") pod \"41df2d2c-e95d-488a-b18f-8862a6239cb3\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") "
Jul 30 02:32:15.399204 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.399024    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41df2d2c-e95d-488a-b18f-8862a6239cb3-config-out\") pod \"41df2d2c-e95d-488a-b18f-8862a6239cb3\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") "
Jul 30 02:32:15.399204 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.399053    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-config-volume\") pod \"41df2d2c-e95d-488a-b18f-8862a6239cb3\" (UID: \"41df2d2c-e95d-488a-b18f-8862a6239cb3\") "
Jul 30 02:32:15.399451 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.399409    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41df2d2c-e95d-488a-b18f-8862a6239cb3-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "41df2d2c-e95d-488a-b18f-8862a6239cb3" (UID: "41df2d2c-e95d-488a-b18f-8862a6239cb3"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 02:32:15.400193 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.399755    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41df2d2c-e95d-488a-b18f-8862a6239cb3-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "41df2d2c-e95d-488a-b18f-8862a6239cb3" (UID: "41df2d2c-e95d-488a-b18f-8862a6239cb3"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:32:15.400193 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.400141    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41df2d2c-e95d-488a-b18f-8862a6239cb3-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "41df2d2c-e95d-488a-b18f-8862a6239cb3" (UID: "41df2d2c-e95d-488a-b18f-8862a6239cb3"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 02:32:15.401941 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.401913    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41df2d2c-e95d-488a-b18f-8862a6239cb3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "41df2d2c-e95d-488a-b18f-8862a6239cb3" (UID: "41df2d2c-e95d-488a-b18f-8862a6239cb3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:32:15.402262 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.402234    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "41df2d2c-e95d-488a-b18f-8862a6239cb3" (UID: "41df2d2c-e95d-488a-b18f-8862a6239cb3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 02:32:15.402358 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.402283    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-config-volume" (OuterVolumeSpecName: "config-volume") pod "41df2d2c-e95d-488a-b18f-8862a6239cb3" (UID: "41df2d2c-e95d-488a-b18f-8862a6239cb3"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 02:32:15.403235 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.403198    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41df2d2c-e95d-488a-b18f-8862a6239cb3-kube-api-access-7ln92" (OuterVolumeSpecName: "kube-api-access-7ln92") pod "41df2d2c-e95d-488a-b18f-8862a6239cb3" (UID: "41df2d2c-e95d-488a-b18f-8862a6239cb3"). InnerVolumeSpecName "kube-api-access-7ln92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:32:15.403348 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.403277    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41df2d2c-e95d-488a-b18f-8862a6239cb3-config-out" (OuterVolumeSpecName: "config-out") pod "41df2d2c-e95d-488a-b18f-8862a6239cb3" (UID: "41df2d2c-e95d-488a-b18f-8862a6239cb3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:32:15.403410 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.403397    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "41df2d2c-e95d-488a-b18f-8862a6239cb3" (UID: "41df2d2c-e95d-488a-b18f-8862a6239cb3"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 02:32:15.403589 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.403571    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "41df2d2c-e95d-488a-b18f-8862a6239cb3" (UID: "41df2d2c-e95d-488a-b18f-8862a6239cb3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 02:32:15.403706 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.403692    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "41df2d2c-e95d-488a-b18f-8862a6239cb3" (UID: "41df2d2c-e95d-488a-b18f-8862a6239cb3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 02:32:15.406310 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.406276    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "41df2d2c-e95d-488a-b18f-8862a6239cb3" (UID: "41df2d2c-e95d-488a-b18f-8862a6239cb3"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 02:32:15.411774 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.411749    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-web-config" (OuterVolumeSpecName: "web-config") pod "41df2d2c-e95d-488a-b18f-8862a6239cb3" (UID: "41df2d2c-e95d-488a-b18f-8862a6239cb3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 02:32:15.499845 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.499818    2565 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-config-volume\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:32:15.499845 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.499842    2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41df2d2c-e95d-488a-b18f-8862a6239cb3-tls-assets\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:32:15.499845 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.499851    2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-web-config\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:32:15.500064 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.499858    2565 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-cluster-tls-config\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:32:15.500064 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.499868    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:32:15.500064 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.499877    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ln92\" (UniqueName: \"kubernetes.io/projected/41df2d2c-e95d-488a-b18f-8862a6239cb3-kube-api-access-7ln92\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:32:15.500064 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.499885    2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41df2d2c-e95d-488a-b18f-8862a6239cb3-alertmanager-main-db\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:32:15.500064 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.499893    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41df2d2c-e95d-488a-b18f-8862a6239cb3-metrics-client-ca\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:32:15.500064 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.499901    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:32:15.500064 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.499910    2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41df2d2c-e95d-488a-b18f-8862a6239cb3-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:32:15.500064 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.499919    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:32:15.500064 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.499928    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41df2d2c-e95d-488a-b18f-8862a6239cb3-secret-alertmanager-main-tls\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:32:15.500064 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:15.499936    2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41df2d2c-e95d-488a-b18f-8862a6239cb3-config-out\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:32:16.010667 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.010633    2565 generic.go:356] "Generic (PLEG): container finished" podID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerID="2df0bc1eeffa2804c75b4950b4f788a98491b117ffbb4a07f144f68fcb30dcfb" exitCode=0
Jul 30 02:32:16.010839 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.010667    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41df2d2c-e95d-488a-b18f-8862a6239cb3","Type":"ContainerDied","Data":"2df0bc1eeffa2804c75b4950b4f788a98491b117ffbb4a07f144f68fcb30dcfb"}
Jul 30 02:32:16.010839 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.010750    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.010839 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.010757    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41df2d2c-e95d-488a-b18f-8862a6239cb3","Type":"ContainerDied","Data":"532acc3ab5a80f2de83803de515572535fbae91468eb7b01be1072a16187a98c"}
Jul 30 02:32:16.011415 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.011396    2565 scope.go:117] "RemoveContainer" containerID="7001017cf0eed65701f29f947559f35be94ff1f344625e4702091fe8739fd535"
Jul 30 02:32:16.019504 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.019488    2565 scope.go:117] "RemoveContainer" containerID="2df0bc1eeffa2804c75b4950b4f788a98491b117ffbb4a07f144f68fcb30dcfb"
Jul 30 02:32:16.026374 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.026354    2565 scope.go:117] "RemoveContainer" containerID="60d862d2860b781a83f2d351359146e1db3068837508caf826b22dac72f0579a"
Jul 30 02:32:16.032837 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.032823    2565 scope.go:117] "RemoveContainer" containerID="698a6fd58c238d2c7c326892367d49a135bb89338c592e94b51dac98f95ded40"
Jul 30 02:32:16.034856 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.034829    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 02:32:16.039537 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.039505    2565 scope.go:117] "RemoveContainer" containerID="1cb14375915dea2a045ec18dbec5eec4c1184479aa49d70ed262b9f338172ad8"
Jul 30 02:32:16.042699 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.042679    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 02:32:16.046380 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.046365    2565 scope.go:117] "RemoveContainer" containerID="9850e8c41f599bfb44c91bc12ee4a909bc73cdf2963956efa3d329b27a7d9e8f"
Jul 30 02:32:16.052950 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.052934    2565 scope.go:117] "RemoveContainer" containerID="b12302717ee2ad8a1b5144ac8cd3c5336dacaa864c6f0f5d10bf377f76e239c3"
Jul 30 02:32:16.059532 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.059517    2565 scope.go:117] "RemoveContainer" containerID="7001017cf0eed65701f29f947559f35be94ff1f344625e4702091fe8739fd535"
Jul 30 02:32:16.059806 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:32:16.059788    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7001017cf0eed65701f29f947559f35be94ff1f344625e4702091fe8739fd535\": container with ID starting with 7001017cf0eed65701f29f947559f35be94ff1f344625e4702091fe8739fd535 not found: ID does not exist" containerID="7001017cf0eed65701f29f947559f35be94ff1f344625e4702091fe8739fd535"
Jul 30 02:32:16.059879 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.059817    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7001017cf0eed65701f29f947559f35be94ff1f344625e4702091fe8739fd535"} err="failed to get container status \"7001017cf0eed65701f29f947559f35be94ff1f344625e4702091fe8739fd535\": rpc error: code = NotFound desc = could not find container \"7001017cf0eed65701f29f947559f35be94ff1f344625e4702091fe8739fd535\": container with ID starting with 7001017cf0eed65701f29f947559f35be94ff1f344625e4702091fe8739fd535 not found: ID does not exist"
Jul 30 02:32:16.059879 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.059841    2565 scope.go:117] "RemoveContainer" containerID="2df0bc1eeffa2804c75b4950b4f788a98491b117ffbb4a07f144f68fcb30dcfb"
Jul 30 02:32:16.060110 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:32:16.060092    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2df0bc1eeffa2804c75b4950b4f788a98491b117ffbb4a07f144f68fcb30dcfb\": container with ID starting with 2df0bc1eeffa2804c75b4950b4f788a98491b117ffbb4a07f144f68fcb30dcfb not found: ID does not exist" containerID="2df0bc1eeffa2804c75b4950b4f788a98491b117ffbb4a07f144f68fcb30dcfb"
Jul 30 02:32:16.060152 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.060116    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df0bc1eeffa2804c75b4950b4f788a98491b117ffbb4a07f144f68fcb30dcfb"} err="failed to get container status \"2df0bc1eeffa2804c75b4950b4f788a98491b117ffbb4a07f144f68fcb30dcfb\": rpc error: code = NotFound desc = could not find container \"2df0bc1eeffa2804c75b4950b4f788a98491b117ffbb4a07f144f68fcb30dcfb\": container with ID starting with 2df0bc1eeffa2804c75b4950b4f788a98491b117ffbb4a07f144f68fcb30dcfb not found: ID does not exist"
Jul 30 02:32:16.060152 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.060131    2565 scope.go:117] "RemoveContainer" containerID="60d862d2860b781a83f2d351359146e1db3068837508caf826b22dac72f0579a"
Jul 30 02:32:16.060393 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:32:16.060375    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60d862d2860b781a83f2d351359146e1db3068837508caf826b22dac72f0579a\": container with ID starting with 60d862d2860b781a83f2d351359146e1db3068837508caf826b22dac72f0579a not found: ID does not exist" containerID="60d862d2860b781a83f2d351359146e1db3068837508caf826b22dac72f0579a"
Jul 30 02:32:16.060434 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.060399    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d862d2860b781a83f2d351359146e1db3068837508caf826b22dac72f0579a"} err="failed to get container status \"60d862d2860b781a83f2d351359146e1db3068837508caf826b22dac72f0579a\": rpc error: code = NotFound desc = could not find container \"60d862d2860b781a83f2d351359146e1db3068837508caf826b22dac72f0579a\": container with ID starting with 60d862d2860b781a83f2d351359146e1db3068837508caf826b22dac72f0579a not found: ID does not exist"
Jul 30 02:32:16.060434 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.060413    2565 scope.go:117] "RemoveContainer" containerID="698a6fd58c238d2c7c326892367d49a135bb89338c592e94b51dac98f95ded40"
Jul 30 02:32:16.060632 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:32:16.060616    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"698a6fd58c238d2c7c326892367d49a135bb89338c592e94b51dac98f95ded40\": container with ID starting with 698a6fd58c238d2c7c326892367d49a135bb89338c592e94b51dac98f95ded40 not found: ID does not exist" containerID="698a6fd58c238d2c7c326892367d49a135bb89338c592e94b51dac98f95ded40"
Jul 30 02:32:16.060681 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.060637    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"698a6fd58c238d2c7c326892367d49a135bb89338c592e94b51dac98f95ded40"} err="failed to get container status \"698a6fd58c238d2c7c326892367d49a135bb89338c592e94b51dac98f95ded40\": rpc error: code = NotFound desc = could not find container \"698a6fd58c238d2c7c326892367d49a135bb89338c592e94b51dac98f95ded40\": container with ID starting with 698a6fd58c238d2c7c326892367d49a135bb89338c592e94b51dac98f95ded40 not found: ID does not exist"
Jul 30 02:32:16.060681 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.060653    2565 scope.go:117] "RemoveContainer" containerID="1cb14375915dea2a045ec18dbec5eec4c1184479aa49d70ed262b9f338172ad8"
Jul 30 02:32:16.060860 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:32:16.060845    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cb14375915dea2a045ec18dbec5eec4c1184479aa49d70ed262b9f338172ad8\": container with ID starting with 1cb14375915dea2a045ec18dbec5eec4c1184479aa49d70ed262b9f338172ad8 not found: ID does not exist" containerID="1cb14375915dea2a045ec18dbec5eec4c1184479aa49d70ed262b9f338172ad8"
Jul 30 02:32:16.060898 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.060864    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cb14375915dea2a045ec18dbec5eec4c1184479aa49d70ed262b9f338172ad8"} err="failed to get container status \"1cb14375915dea2a045ec18dbec5eec4c1184479aa49d70ed262b9f338172ad8\": rpc error: code = NotFound desc = could not find container \"1cb14375915dea2a045ec18dbec5eec4c1184479aa49d70ed262b9f338172ad8\": container with ID starting with 1cb14375915dea2a045ec18dbec5eec4c1184479aa49d70ed262b9f338172ad8 not found: ID does not exist"
Jul 30 02:32:16.060898 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.060876    2565 scope.go:117] "RemoveContainer" containerID="9850e8c41f599bfb44c91bc12ee4a909bc73cdf2963956efa3d329b27a7d9e8f"
Jul 30 02:32:16.061064 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:32:16.061048    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9850e8c41f599bfb44c91bc12ee4a909bc73cdf2963956efa3d329b27a7d9e8f\": container with ID starting with 9850e8c41f599bfb44c91bc12ee4a909bc73cdf2963956efa3d329b27a7d9e8f not found: ID does not exist" containerID="9850e8c41f599bfb44c91bc12ee4a909bc73cdf2963956efa3d329b27a7d9e8f"
Jul 30 02:32:16.061101 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.061067    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9850e8c41f599bfb44c91bc12ee4a909bc73cdf2963956efa3d329b27a7d9e8f"} err="failed to get container status \"9850e8c41f599bfb44c91bc12ee4a909bc73cdf2963956efa3d329b27a7d9e8f\": rpc error: code = NotFound desc = could not find container \"9850e8c41f599bfb44c91bc12ee4a909bc73cdf2963956efa3d329b27a7d9e8f\": container with ID starting with 9850e8c41f599bfb44c91bc12ee4a909bc73cdf2963956efa3d329b27a7d9e8f not found: ID does not exist"
Jul 30 02:32:16.061101 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.061079    2565 scope.go:117] "RemoveContainer" containerID="b12302717ee2ad8a1b5144ac8cd3c5336dacaa864c6f0f5d10bf377f76e239c3"
Jul 30 02:32:16.061292 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:32:16.061276    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b12302717ee2ad8a1b5144ac8cd3c5336dacaa864c6f0f5d10bf377f76e239c3\": container with ID starting with b12302717ee2ad8a1b5144ac8cd3c5336dacaa864c6f0f5d10bf377f76e239c3 not found: ID does not exist" containerID="b12302717ee2ad8a1b5144ac8cd3c5336dacaa864c6f0f5d10bf377f76e239c3"
Jul 30 02:32:16.061331 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.061297    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b12302717ee2ad8a1b5144ac8cd3c5336dacaa864c6f0f5d10bf377f76e239c3"} err="failed to get container status \"b12302717ee2ad8a1b5144ac8cd3c5336dacaa864c6f0f5d10bf377f76e239c3\": rpc error: code = NotFound desc = could not find container \"b12302717ee2ad8a1b5144ac8cd3c5336dacaa864c6f0f5d10bf377f76e239c3\": container with ID starting with b12302717ee2ad8a1b5144ac8cd3c5336dacaa864c6f0f5d10bf377f76e239c3 not found: ID does not exist"
Jul 30 02:32:16.070767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.070746    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 02:32:16.070993 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.070980    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="init-config-reloader"
Jul 30 02:32:16.071032 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.070997    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="init-config-reloader"
Jul 30 02:32:16.071032 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.071005    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="kube-rbac-proxy-metric"
Jul 30 02:32:16.071032 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.071011    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="kube-rbac-proxy-metric"
Jul 30 02:32:16.071032 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.071017    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="kube-rbac-proxy-web"
Jul 30 02:32:16.071032 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.071024    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="kube-rbac-proxy-web"
Jul 30 02:32:16.071032 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.071029    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="kube-rbac-proxy"
Jul 30 02:32:16.071032 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.071034    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="kube-rbac-proxy"
Jul 30 02:32:16.071249 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.071041    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="config-reloader"
Jul 30 02:32:16.071249 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.071046    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="config-reloader"
Jul 30 02:32:16.071249 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.071053    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="prom-label-proxy"
Jul 30 02:32:16.071249 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.071058    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="prom-label-proxy"
Jul 30 02:32:16.071249 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.071067    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="alertmanager"
Jul 30 02:32:16.071249 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.071076    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="alertmanager"
Jul 30 02:32:16.071249 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.071120    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="alertmanager"
Jul 30 02:32:16.071249 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.071128    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="kube-rbac-proxy-web"
Jul 30 02:32:16.071249 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.071135    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="kube-rbac-proxy"
Jul 30 02:32:16.071249 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.071142    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="prom-label-proxy"
Jul 30 02:32:16.071249 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.071152    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="config-reloader"
Jul 30 02:32:16.071249 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.071157    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" containerName="kube-rbac-proxy-metric"
Jul 30 02:32:16.074383 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.074369    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.077234 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.077211    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 30 02:32:16.077234 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.077225    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xfjt2\""
Jul 30 02:32:16.077382 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.077211    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 30 02:32:16.077382 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.077291    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 30 02:32:16.077485 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.077473    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 30 02:32:16.077536 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.077511    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 30 02:32:16.077627 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.077606    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 30 02:32:16.077686 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.077624    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 30 02:32:16.077686 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.077613    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 30 02:32:16.083917 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.083898    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 30 02:32:16.092774 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.092752    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 02:32:16.205190 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.205133    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/13da4179-ab3f-40d0-b08b-0978fce78314-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.205379 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.205218    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13da4179-ab3f-40d0-b08b-0978fce78314-config-out\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.205379 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.205245    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-web-config\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.205379 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.205302    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-config-volume\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.205379 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.205332    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.205379 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.205368    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.205636 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.205423    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13da4179-ab3f-40d0-b08b-0978fce78314-tls-assets\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.205636 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.205446    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13da4179-ab3f-40d0-b08b-0978fce78314-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.205636 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.205470    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.205636 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.205495    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psqz6\" (UniqueName: \"kubernetes.io/projected/13da4179-ab3f-40d0-b08b-0978fce78314-kube-api-access-psqz6\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.205636 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.205521    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13da4179-ab3f-40d0-b08b-0978fce78314-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.205636 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.205562    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.205636 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.205595    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.306573 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.306543    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-config-volume\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.306996 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.306577    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.306996 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.306597    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.306996 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.306616    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13da4179-ab3f-40d0-b08b-0978fce78314-tls-assets\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.306996 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.306632    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13da4179-ab3f-40d0-b08b-0978fce78314-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.306996 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.306647    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.306996 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.306663    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psqz6\" (UniqueName: \"kubernetes.io/projected/13da4179-ab3f-40d0-b08b-0978fce78314-kube-api-access-psqz6\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.306996 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.306687    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13da4179-ab3f-40d0-b08b-0978fce78314-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.306996 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.306717    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.306996 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.306741    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.306996 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.306774    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/13da4179-ab3f-40d0-b08b-0978fce78314-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.306996 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.306815    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13da4179-ab3f-40d0-b08b-0978fce78314-config-out\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.306996 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.306838    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-web-config\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.307648 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.307634    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13da4179-ab3f-40d0-b08b-0978fce78314-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.308437 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.308042    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13da4179-ab3f-40d0-b08b-0978fce78314-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.308437 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.308375    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/13da4179-ab3f-40d0-b08b-0978fce78314-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.310361 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.310216    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.310361 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.310247    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.310361 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.310309    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-config-volume\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.310633 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.310604    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.310633 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.310616    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13da4179-ab3f-40d0-b08b-0978fce78314-tls-assets\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.310754 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.310642    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.310938 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.310916    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-web-config\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.311589 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.311571    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13da4179-ab3f-40d0-b08b-0978fce78314-config-out\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.311696 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.311677    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/13da4179-ab3f-40d0-b08b-0978fce78314-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.315319 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.315295    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psqz6\" (UniqueName: \"kubernetes.io/projected/13da4179-ab3f-40d0-b08b-0978fce78314-kube-api-access-psqz6\") pod \"alertmanager-main-0\" (UID: \"13da4179-ab3f-40d0-b08b-0978fce78314\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.384718 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.384689    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 30 02:32:16.512578 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.512547    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 02:32:16.515625 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:32:16.515593    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13da4179_ab3f_40d0_b08b_0978fce78314.slice/crio-7fe10de296e0ccac5dceb55ce60f340bb1990c111b0f2e25c274139ead5bc7bc WatchSource:0}: Error finding container 7fe10de296e0ccac5dceb55ce60f340bb1990c111b0f2e25c274139ead5bc7bc: Status 404 returned error can't find the container with id 7fe10de296e0ccac5dceb55ce60f340bb1990c111b0f2e25c274139ead5bc7bc
Jul 30 02:32:16.799792 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:16.799757    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41df2d2c-e95d-488a-b18f-8862a6239cb3" path="/var/lib/kubelet/pods/41df2d2c-e95d-488a-b18f-8862a6239cb3/volumes"
Jul 30 02:32:17.015566 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:17.015483    2565 generic.go:356] "Generic (PLEG): container finished" podID="13da4179-ab3f-40d0-b08b-0978fce78314" containerID="7791beb01fa49fcf95323e63fe87f0a728240a690b66b34c8901468480fa4b2d" exitCode=0
Jul 30 02:32:17.015729 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:17.015576    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13da4179-ab3f-40d0-b08b-0978fce78314","Type":"ContainerDied","Data":"7791beb01fa49fcf95323e63fe87f0a728240a690b66b34c8901468480fa4b2d"}
Jul 30 02:32:17.015729 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:17.015654    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13da4179-ab3f-40d0-b08b-0978fce78314","Type":"ContainerStarted","Data":"7fe10de296e0ccac5dceb55ce60f340bb1990c111b0f2e25c274139ead5bc7bc"}
Jul 30 02:32:17.616201 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:17.616142    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs\") pod \"network-metrics-daemon-wphj8\" (UID: \"ec397db7-4d16-4a1d-9ace-fb4de6e17039\") " pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:32:17.618629 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:17.618603    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec397db7-4d16-4a1d-9ace-fb4de6e17039-metrics-certs\") pod \"network-metrics-daemon-wphj8\" (UID: \"ec397db7-4d16-4a1d-9ace-fb4de6e17039\") " pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:32:17.800293 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:17.800263    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vvjjn\""
Jul 30 02:32:17.808751 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:17.808716    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wphj8"
Jul 30 02:32:17.933437 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:17.931494    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wphj8"]
Jul 30 02:32:17.933926 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:32:17.933903    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec397db7_4d16_4a1d_9ace_fb4de6e17039.slice/crio-fc4b10c95c6037958e49176f78ad7d4ef3a841c09bd2ceeced35c7e2b6680382 WatchSource:0}: Error finding container fc4b10c95c6037958e49176f78ad7d4ef3a841c09bd2ceeced35c7e2b6680382: Status 404 returned error can't find the container with id fc4b10c95c6037958e49176f78ad7d4ef3a841c09bd2ceeced35c7e2b6680382
Jul 30 02:32:17.935955 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:17.935936    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 02:32:18.023320 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:18.023285    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13da4179-ab3f-40d0-b08b-0978fce78314","Type":"ContainerStarted","Data":"941cdeae9591c6475ebee43dd719f996d5a33bc5ccd1236e38db42467534db48"}
Jul 30 02:32:18.023490 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:18.023360    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13da4179-ab3f-40d0-b08b-0978fce78314","Type":"ContainerStarted","Data":"33a3db87306cf558369fe251feab652f5f6d05e32ba6def918b12d56c88768d3"}
Jul 30 02:32:18.023490 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:18.023389    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13da4179-ab3f-40d0-b08b-0978fce78314","Type":"ContainerStarted","Data":"c09e99d79bdae46639a5ced1c8cd0a7f5a628109cb0823a7ab0d257295d1f5ad"}
Jul 30 02:32:18.023490 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:18.023415    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13da4179-ab3f-40d0-b08b-0978fce78314","Type":"ContainerStarted","Data":"1995ac716c3d4925da83c23514eea75d2599b10c01fe5101677b0f9737255ea1"}
Jul 30 02:32:18.023490 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:18.023449    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13da4179-ab3f-40d0-b08b-0978fce78314","Type":"ContainerStarted","Data":"58169ef8121ff4f438ddca000c65bd05aecd806dff2a04c49a38935c5f04e0fb"}
Jul 30 02:32:18.023490 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:18.023474    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13da4179-ab3f-40d0-b08b-0978fce78314","Type":"ContainerStarted","Data":"1c5437d9865507bcc679be2d719ca054da211e7b7643c1e65082f50afd9021b5"}
Jul 30 02:32:18.024468 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:18.024448    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wphj8" event={"ID":"ec397db7-4d16-4a1d-9ace-fb4de6e17039","Type":"ContainerStarted","Data":"fc4b10c95c6037958e49176f78ad7d4ef3a841c09bd2ceeced35c7e2b6680382"}
Jul 30 02:32:18.053716 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:18.053670    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.034375531 podStartE2EDuration="2.05365614s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.280609ms" isStatefulPod=false podCreationTimestamp="2026-07-30 02:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:32:18.051418565 +0000 UTC m=+373.698201970" watchObservedRunningTime="2026-07-30 02:32:18.05365614 +0000 UTC m=+373.700439544"
Jul 30 02:32:19.030154 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:19.030102    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wphj8" event={"ID":"ec397db7-4d16-4a1d-9ace-fb4de6e17039","Type":"ContainerStarted","Data":"511eba01c77ac827aacdc17afb8b639d84427b6693e917be0eb6143258fda6e5"}
Jul 30 02:32:19.030536 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:19.030186    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wphj8" event={"ID":"ec397db7-4d16-4a1d-9ace-fb4de6e17039","Type":"ContainerStarted","Data":"ed749c55df47f207d9950538e3d3308e0e125f6c37bc2331c23562c53e2c9445"}
Jul 30 02:32:19.048667 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:32:19.048586    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wphj8" podStartSLOduration=373.193021232 podStartE2EDuration="6m14.048571372s" totalImagesPullingTime="855.55014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:32:19.047128232 +0000 UTC m=+374.693911636" watchObservedRunningTime="2026-07-30 02:32:19.048571372 +0000 UTC m=+374.695354783"
Jul 30 02:33:15.313041 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:15.312998    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f"]
Jul 30 02:33:15.317688 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:15.317666    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f"
Jul 30 02:33:15.320536 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:15.320516    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 30 02:33:15.321736 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:15.321719    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 30 02:33:15.321788 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:15.321745    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7sbvw\""
Jul 30 02:33:15.325747 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:15.325723    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f"]
Jul 30 02:33:15.436047 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:15.436010    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfwx9\" (UniqueName: \"kubernetes.io/projected/6c51d78c-70a1-4c00-bc6d-24d800a8e464-kube-api-access-sfwx9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f\" (UID: \"6c51d78c-70a1-4c00-bc6d-24d800a8e464\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f"
Jul 30 02:33:15.436047 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:15.436053    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c51d78c-70a1-4c00-bc6d-24d800a8e464-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f\" (UID: \"6c51d78c-70a1-4c00-bc6d-24d800a8e464\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f"
Jul 30 02:33:15.436301 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:15.436077    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c51d78c-70a1-4c00-bc6d-24d800a8e464-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f\" (UID: \"6c51d78c-70a1-4c00-bc6d-24d800a8e464\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f"
Jul 30 02:33:15.537142 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:15.537094    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfwx9\" (UniqueName: \"kubernetes.io/projected/6c51d78c-70a1-4c00-bc6d-24d800a8e464-kube-api-access-sfwx9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f\" (UID: \"6c51d78c-70a1-4c00-bc6d-24d800a8e464\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f"
Jul 30 02:33:15.537142 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:15.537149    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c51d78c-70a1-4c00-bc6d-24d800a8e464-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f\" (UID: \"6c51d78c-70a1-4c00-bc6d-24d800a8e464\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f"
Jul 30 02:33:15.537377 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:15.537186    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c51d78c-70a1-4c00-bc6d-24d800a8e464-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f\" (UID: \"6c51d78c-70a1-4c00-bc6d-24d800a8e464\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f"
Jul 30 02:33:15.537559 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:15.537539    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c51d78c-70a1-4c00-bc6d-24d800a8e464-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f\" (UID: \"6c51d78c-70a1-4c00-bc6d-24d800a8e464\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f"
Jul 30 02:33:15.537559 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:15.537551    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c51d78c-70a1-4c00-bc6d-24d800a8e464-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f\" (UID: \"6c51d78c-70a1-4c00-bc6d-24d800a8e464\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f"
Jul 30 02:33:15.545823 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:15.545805    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfwx9\" (UniqueName: \"kubernetes.io/projected/6c51d78c-70a1-4c00-bc6d-24d800a8e464-kube-api-access-sfwx9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f\" (UID: \"6c51d78c-70a1-4c00-bc6d-24d800a8e464\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f"
Jul 30 02:33:15.627226 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:15.627146    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f"
Jul 30 02:33:15.755366 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:15.755338    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f"]
Jul 30 02:33:15.757204 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:33:15.757162    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c51d78c_70a1_4c00_bc6d_24d800a8e464.slice/crio-3ea4b17fd7631a24e815dfa095aabb87bb1bddebef0bc2f665189a1603db1853 WatchSource:0}: Error finding container 3ea4b17fd7631a24e815dfa095aabb87bb1bddebef0bc2f665189a1603db1853: Status 404 returned error can't find the container with id 3ea4b17fd7631a24e815dfa095aabb87bb1bddebef0bc2f665189a1603db1853
Jul 30 02:33:16.231385 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:16.231350    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f" event={"ID":"6c51d78c-70a1-4c00-bc6d-24d800a8e464","Type":"ContainerStarted","Data":"3ea4b17fd7631a24e815dfa095aabb87bb1bddebef0bc2f665189a1603db1853"}
Jul 30 02:33:23.259952 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:23.259860    2565 generic.go:356] "Generic (PLEG): container finished" podID="6c51d78c-70a1-4c00-bc6d-24d800a8e464" containerID="d7f6afc9a7195596160ce9a2879cbc6813a5631e5b0354a09a560548b5b42bed" exitCode=0
Jul 30 02:33:23.260333 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:23.259945    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f" event={"ID":"6c51d78c-70a1-4c00-bc6d-24d800a8e464","Type":"ContainerDied","Data":"d7f6afc9a7195596160ce9a2879cbc6813a5631e5b0354a09a560548b5b42bed"}
Jul 30 02:33:26.275155 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:26.275122    2565 generic.go:356] "Generic (PLEG): container finished" podID="6c51d78c-70a1-4c00-bc6d-24d800a8e464" containerID="afd397b20d8e50580bac80f88648ed1f9ba08757767dd783eea0027b7c6bba80" exitCode=0
Jul 30 02:33:26.275539 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:26.275216    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f" event={"ID":"6c51d78c-70a1-4c00-bc6d-24d800a8e464","Type":"ContainerDied","Data":"afd397b20d8e50580bac80f88648ed1f9ba08757767dd783eea0027b7c6bba80"}
Jul 30 02:33:34.309080 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:34.309041    2565 generic.go:356] "Generic (PLEG): container finished" podID="6c51d78c-70a1-4c00-bc6d-24d800a8e464" containerID="bee6c1a1e89ac7cda6822aaa3bd676a6426b67333192c25aceb9f7288cce1c47" exitCode=0
Jul 30 02:33:34.309458 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:34.309127    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f" event={"ID":"6c51d78c-70a1-4c00-bc6d-24d800a8e464","Type":"ContainerDied","Data":"bee6c1a1e89ac7cda6822aaa3bd676a6426b67333192c25aceb9f7288cce1c47"}
Jul 30 02:33:35.444378 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:35.444352    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f"
Jul 30 02:33:35.611700 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:35.611622    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfwx9\" (UniqueName: \"kubernetes.io/projected/6c51d78c-70a1-4c00-bc6d-24d800a8e464-kube-api-access-sfwx9\") pod \"6c51d78c-70a1-4c00-bc6d-24d800a8e464\" (UID: \"6c51d78c-70a1-4c00-bc6d-24d800a8e464\") "
Jul 30 02:33:35.611700 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:35.611660    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c51d78c-70a1-4c00-bc6d-24d800a8e464-util\") pod \"6c51d78c-70a1-4c00-bc6d-24d800a8e464\" (UID: \"6c51d78c-70a1-4c00-bc6d-24d800a8e464\") "
Jul 30 02:33:35.611966 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:35.611736    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c51d78c-70a1-4c00-bc6d-24d800a8e464-bundle\") pod \"6c51d78c-70a1-4c00-bc6d-24d800a8e464\" (UID: \"6c51d78c-70a1-4c00-bc6d-24d800a8e464\") "
Jul 30 02:33:35.612407 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:35.612387    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c51d78c-70a1-4c00-bc6d-24d800a8e464-bundle" (OuterVolumeSpecName: "bundle") pod "6c51d78c-70a1-4c00-bc6d-24d800a8e464" (UID: "6c51d78c-70a1-4c00-bc6d-24d800a8e464"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:33:35.613930 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:35.613905    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c51d78c-70a1-4c00-bc6d-24d800a8e464-kube-api-access-sfwx9" (OuterVolumeSpecName: "kube-api-access-sfwx9") pod "6c51d78c-70a1-4c00-bc6d-24d800a8e464" (UID: "6c51d78c-70a1-4c00-bc6d-24d800a8e464"). InnerVolumeSpecName "kube-api-access-sfwx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:33:35.615686 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:35.615663    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c51d78c-70a1-4c00-bc6d-24d800a8e464-util" (OuterVolumeSpecName: "util") pod "6c51d78c-70a1-4c00-bc6d-24d800a8e464" (UID: "6c51d78c-70a1-4c00-bc6d-24d800a8e464"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:33:35.712449 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:35.712418    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c51d78c-70a1-4c00-bc6d-24d800a8e464-bundle\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:33:35.712449 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:35.712446    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfwx9\" (UniqueName: \"kubernetes.io/projected/6c51d78c-70a1-4c00-bc6d-24d800a8e464-kube-api-access-sfwx9\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:33:35.712634 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:35.712458    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c51d78c-70a1-4c00-bc6d-24d800a8e464-util\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:33:36.317720 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:36.317688    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f" event={"ID":"6c51d78c-70a1-4c00-bc6d-24d800a8e464","Type":"ContainerDied","Data":"3ea4b17fd7631a24e815dfa095aabb87bb1bddebef0bc2f665189a1603db1853"}
Jul 30 02:33:36.317720 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:36.317721    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rq64f"
Jul 30 02:33:36.317918 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:36.317731    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ea4b17fd7631a24e815dfa095aabb87bb1bddebef0bc2f665189a1603db1853"
Jul 30 02:33:48.012136 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.012107    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv"]
Jul 30 02:33:48.012510 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.012365    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c51d78c-70a1-4c00-bc6d-24d800a8e464" containerName="util"
Jul 30 02:33:48.012510 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.012375    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c51d78c-70a1-4c00-bc6d-24d800a8e464" containerName="util"
Jul 30 02:33:48.012510 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.012393    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c51d78c-70a1-4c00-bc6d-24d800a8e464" containerName="pull"
Jul 30 02:33:48.012510 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.012398    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c51d78c-70a1-4c00-bc6d-24d800a8e464" containerName="pull"
Jul 30 02:33:48.012510 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.012404    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c51d78c-70a1-4c00-bc6d-24d800a8e464" containerName="extract"
Jul 30 02:33:48.012510 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.012409    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c51d78c-70a1-4c00-bc6d-24d800a8e464" containerName="extract"
Jul 30 02:33:48.012510 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.012455    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c51d78c-70a1-4c00-bc6d-24d800a8e464" containerName="extract"
Jul 30 02:33:48.015517 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.015502    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv"
Jul 30 02:33:48.019056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.019023    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7sbvw\""
Jul 30 02:33:48.019208 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.019067    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 30 02:33:48.020285 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.020267    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 30 02:33:48.031765 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.031739    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv"]
Jul 30 02:33:48.197203 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.197153    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc6c0daf-a843-4583-8f97-37748627ebf7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv\" (UID: \"dc6c0daf-a843-4583-8f97-37748627ebf7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv"
Jul 30 02:33:48.197367 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.197215    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc6c0daf-a843-4583-8f97-37748627ebf7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv\" (UID: \"dc6c0daf-a843-4583-8f97-37748627ebf7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv"
Jul 30 02:33:48.197367 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.197255    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdmzd\" (UniqueName: \"kubernetes.io/projected/dc6c0daf-a843-4583-8f97-37748627ebf7-kube-api-access-sdmzd\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv\" (UID: \"dc6c0daf-a843-4583-8f97-37748627ebf7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv"
Jul 30 02:33:48.297912 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.297887    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc6c0daf-a843-4583-8f97-37748627ebf7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv\" (UID: \"dc6c0daf-a843-4583-8f97-37748627ebf7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv"
Jul 30 02:33:48.298048 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.297925    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmzd\" (UniqueName: \"kubernetes.io/projected/dc6c0daf-a843-4583-8f97-37748627ebf7-kube-api-access-sdmzd\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv\" (UID: \"dc6c0daf-a843-4583-8f97-37748627ebf7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv"
Jul 30 02:33:48.298048 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.297985    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc6c0daf-a843-4583-8f97-37748627ebf7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv\" (UID: \"dc6c0daf-a843-4583-8f97-37748627ebf7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv"
Jul 30 02:33:48.298375 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.298354    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc6c0daf-a843-4583-8f97-37748627ebf7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv\" (UID: \"dc6c0daf-a843-4583-8f97-37748627ebf7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv"
Jul 30 02:33:48.298419 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.298381    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc6c0daf-a843-4583-8f97-37748627ebf7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv\" (UID: \"dc6c0daf-a843-4583-8f97-37748627ebf7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv"
Jul 30 02:33:48.306401 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.306382    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdmzd\" (UniqueName: \"kubernetes.io/projected/dc6c0daf-a843-4583-8f97-37748627ebf7-kube-api-access-sdmzd\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv\" (UID: \"dc6c0daf-a843-4583-8f97-37748627ebf7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv"
Jul 30 02:33:48.323702 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.323679    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv"
Jul 30 02:33:48.446721 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:48.446697    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv"]
Jul 30 02:33:48.449393 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:33:48.449365    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6c0daf_a843_4583_8f97_37748627ebf7.slice/crio-1799e8a646b4b131b235e91725e770b4e003e09ab7275ee3fab3ccbc5cf49e4c WatchSource:0}: Error finding container 1799e8a646b4b131b235e91725e770b4e003e09ab7275ee3fab3ccbc5cf49e4c: Status 404 returned error can't find the container with id 1799e8a646b4b131b235e91725e770b4e003e09ab7275ee3fab3ccbc5cf49e4c
Jul 30 02:33:49.368081 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:49.368047    2565 generic.go:356] "Generic (PLEG): container finished" podID="dc6c0daf-a843-4583-8f97-37748627ebf7" containerID="af7e012d46036c293309735d8ac1f948088865d23bf79c282d2d98dca70fb619" exitCode=0
Jul 30 02:33:49.368525 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:49.368141    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv" event={"ID":"dc6c0daf-a843-4583-8f97-37748627ebf7","Type":"ContainerDied","Data":"af7e012d46036c293309735d8ac1f948088865d23bf79c282d2d98dca70fb619"}
Jul 30 02:33:49.368525 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:49.368225    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv" event={"ID":"dc6c0daf-a843-4583-8f97-37748627ebf7","Type":"ContainerStarted","Data":"1799e8a646b4b131b235e91725e770b4e003e09ab7275ee3fab3ccbc5cf49e4c"}
Jul 30 02:33:55.391896 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:55.391865    2565 generic.go:356] "Generic (PLEG): container finished" podID="dc6c0daf-a843-4583-8f97-37748627ebf7" containerID="34bd7ce85faf7df33597d4c2b68de88239f0e3e08f5b8e4b63d5e99be37e9065" exitCode=0
Jul 30 02:33:55.392321 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:55.391954    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv" event={"ID":"dc6c0daf-a843-4583-8f97-37748627ebf7","Type":"ContainerDied","Data":"34bd7ce85faf7df33597d4c2b68de88239f0e3e08f5b8e4b63d5e99be37e9065"}
Jul 30 02:33:56.397674 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:56.397635    2565 generic.go:356] "Generic (PLEG): container finished" podID="dc6c0daf-a843-4583-8f97-37748627ebf7" containerID="a96d718f6321c5a1800d693303bc51bae4ace73cb21e9a2df5f84e9c0fb96286" exitCode=0
Jul 30 02:33:56.398059 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:56.397718    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv" event={"ID":"dc6c0daf-a843-4583-8f97-37748627ebf7","Type":"ContainerDied","Data":"a96d718f6321c5a1800d693303bc51bae4ace73cb21e9a2df5f84e9c0fb96286"}
Jul 30 02:33:57.522405 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:57.522378    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv"
Jul 30 02:33:57.568391 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:57.568363    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc6c0daf-a843-4583-8f97-37748627ebf7-util\") pod \"dc6c0daf-a843-4583-8f97-37748627ebf7\" (UID: \"dc6c0daf-a843-4583-8f97-37748627ebf7\") "
Jul 30 02:33:57.568391 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:57.568390    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc6c0daf-a843-4583-8f97-37748627ebf7-bundle\") pod \"dc6c0daf-a843-4583-8f97-37748627ebf7\" (UID: \"dc6c0daf-a843-4583-8f97-37748627ebf7\") "
Jul 30 02:33:57.568614 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:57.568422    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdmzd\" (UniqueName: \"kubernetes.io/projected/dc6c0daf-a843-4583-8f97-37748627ebf7-kube-api-access-sdmzd\") pod \"dc6c0daf-a843-4583-8f97-37748627ebf7\" (UID: \"dc6c0daf-a843-4583-8f97-37748627ebf7\") "
Jul 30 02:33:57.568790 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:57.568766    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6c0daf-a843-4583-8f97-37748627ebf7-bundle" (OuterVolumeSpecName: "bundle") pod "dc6c0daf-a843-4583-8f97-37748627ebf7" (UID: "dc6c0daf-a843-4583-8f97-37748627ebf7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:33:57.570580 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:57.570559    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6c0daf-a843-4583-8f97-37748627ebf7-kube-api-access-sdmzd" (OuterVolumeSpecName: "kube-api-access-sdmzd") pod "dc6c0daf-a843-4583-8f97-37748627ebf7" (UID: "dc6c0daf-a843-4583-8f97-37748627ebf7"). InnerVolumeSpecName "kube-api-access-sdmzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:33:57.572878 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:57.572856    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6c0daf-a843-4583-8f97-37748627ebf7-util" (OuterVolumeSpecName: "util") pod "dc6c0daf-a843-4583-8f97-37748627ebf7" (UID: "dc6c0daf-a843-4583-8f97-37748627ebf7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:33:57.669461 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:57.669386    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdmzd\" (UniqueName: \"kubernetes.io/projected/dc6c0daf-a843-4583-8f97-37748627ebf7-kube-api-access-sdmzd\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:33:57.669461 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:57.669412    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc6c0daf-a843-4583-8f97-37748627ebf7-util\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:33:57.669461 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:57.669424    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc6c0daf-a843-4583-8f97-37748627ebf7-bundle\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:33:58.406342 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:58.406317    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv"
Jul 30 02:33:58.406342 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:58.406322    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwltsv" event={"ID":"dc6c0daf-a843-4583-8f97-37748627ebf7","Type":"ContainerDied","Data":"1799e8a646b4b131b235e91725e770b4e003e09ab7275ee3fab3ccbc5cf49e4c"}
Jul 30 02:33:58.406537 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:33:58.406366    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1799e8a646b4b131b235e91725e770b4e003e09ab7275ee3fab3ccbc5cf49e4c"
Jul 30 02:34:01.213644 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.213612    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5blr2"]
Jul 30 02:34:01.214017 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.213891    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc6c0daf-a843-4583-8f97-37748627ebf7" containerName="extract"
Jul 30 02:34:01.214017 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.213902    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc6c0daf-a843-4583-8f97-37748627ebf7" containerName="extract"
Jul 30 02:34:01.214017 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.213915    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc6c0daf-a843-4583-8f97-37748627ebf7" containerName="pull"
Jul 30 02:34:01.214017 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.213920    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc6c0daf-a843-4583-8f97-37748627ebf7" containerName="pull"
Jul 30 02:34:01.214017 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.213927    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc6c0daf-a843-4583-8f97-37748627ebf7" containerName="util"
Jul 30 02:34:01.214017 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.213932    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc6c0daf-a843-4583-8f97-37748627ebf7" containerName="util"
Jul 30 02:34:01.214017 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.213993    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc6c0daf-a843-4583-8f97-37748627ebf7" containerName="extract"
Jul 30 02:34:01.216784 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.216768    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5blr2"
Jul 30 02:34:01.219987 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.219958    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 30 02:34:01.220128 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.220003    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-c27sk\""
Jul 30 02:34:01.221882 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.221863    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 30 02:34:01.226289 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.226263    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5blr2"]
Jul 30 02:34:01.299896 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.299857    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f46c6b01-8d08-4631-bfdb-523159676aad-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5blr2\" (UID: \"f46c6b01-8d08-4631-bfdb-523159676aad\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5blr2"
Jul 30 02:34:01.300073 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.299919    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhwbl\" (UniqueName: \"kubernetes.io/projected/f46c6b01-8d08-4631-bfdb-523159676aad-kube-api-access-mhwbl\") pod \"openshift-lws-operator-fd8ccff4c-5blr2\" (UID: \"f46c6b01-8d08-4631-bfdb-523159676aad\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5blr2"
Jul 30 02:34:01.400828 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.400789    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f46c6b01-8d08-4631-bfdb-523159676aad-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5blr2\" (UID: \"f46c6b01-8d08-4631-bfdb-523159676aad\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5blr2"
Jul 30 02:34:01.401016 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.400841    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhwbl\" (UniqueName: \"kubernetes.io/projected/f46c6b01-8d08-4631-bfdb-523159676aad-kube-api-access-mhwbl\") pod \"openshift-lws-operator-fd8ccff4c-5blr2\" (UID: \"f46c6b01-8d08-4631-bfdb-523159676aad\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5blr2"
Jul 30 02:34:01.401288 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.401266    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f46c6b01-8d08-4631-bfdb-523159676aad-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5blr2\" (UID: \"f46c6b01-8d08-4631-bfdb-523159676aad\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5blr2"
Jul 30 02:34:01.410072 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.410048    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhwbl\" (UniqueName: \"kubernetes.io/projected/f46c6b01-8d08-4631-bfdb-523159676aad-kube-api-access-mhwbl\") pod \"openshift-lws-operator-fd8ccff4c-5blr2\" (UID: \"f46c6b01-8d08-4631-bfdb-523159676aad\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5blr2"
Jul 30 02:34:01.527192 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.527074    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5blr2"
Jul 30 02:34:01.650330 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:01.650299    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5blr2"]
Jul 30 02:34:01.653424 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:34:01.653392    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf46c6b01_8d08_4631_bfdb_523159676aad.slice/crio-707fcef595306a394ddc1503560f97315e91681e184495aea2222d81ba62c02b WatchSource:0}: Error finding container 707fcef595306a394ddc1503560f97315e91681e184495aea2222d81ba62c02b: Status 404 returned error can't find the container with id 707fcef595306a394ddc1503560f97315e91681e184495aea2222d81ba62c02b
Jul 30 02:34:02.422495 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:02.422458    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5blr2" event={"ID":"f46c6b01-8d08-4631-bfdb-523159676aad","Type":"ContainerStarted","Data":"707fcef595306a394ddc1503560f97315e91681e184495aea2222d81ba62c02b"}
Jul 30 02:34:04.431976 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:04.431938    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5blr2" event={"ID":"f46c6b01-8d08-4631-bfdb-523159676aad","Type":"ContainerStarted","Data":"a43e99a375b571ae9b980aa7b5b63c523fe95d76faa015e95d562d3ffbbb0d26"}
Jul 30 02:34:04.449688 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:04.449638    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5blr2" podStartSLOduration=0.829756206 podStartE2EDuration="3.449624418s" totalImagesPullingTime="2.619868212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:34:04.447505912 +0000 UTC m=+480.094289309" watchObservedRunningTime="2026-07-30 02:34:04.449624418 +0000 UTC m=+480.096407824"
Jul 30 02:34:07.032680 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:07.032644    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc"]
Jul 30 02:34:07.051830 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:07.051806    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc"]
Jul 30 02:34:07.051977 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:07.051910    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc"
Jul 30 02:34:07.054854 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:07.054833    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 30 02:34:07.054854 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:07.054849    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 30 02:34:07.056015 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:07.055997    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7sbvw\""
Jul 30 02:34:07.142036 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:07.142005    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50f91d8e-778f-4eea-a94b-fe231b0922d2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc\" (UID: \"50f91d8e-778f-4eea-a94b-fe231b0922d2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc"
Jul 30 02:34:07.142036 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:07.142034    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50f91d8e-778f-4eea-a94b-fe231b0922d2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc\" (UID: \"50f91d8e-778f-4eea-a94b-fe231b0922d2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc"
Jul 30 02:34:07.142259 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:07.142131    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr9px\" (UniqueName: \"kubernetes.io/projected/50f91d8e-778f-4eea-a94b-fe231b0922d2-kube-api-access-zr9px\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc\" (UID: \"50f91d8e-778f-4eea-a94b-fe231b0922d2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc"
Jul 30 02:34:07.243580 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:07.243550    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr9px\" (UniqueName: \"kubernetes.io/projected/50f91d8e-778f-4eea-a94b-fe231b0922d2-kube-api-access-zr9px\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc\" (UID: \"50f91d8e-778f-4eea-a94b-fe231b0922d2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc"
Jul 30 02:34:07.243748 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:07.243591    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50f91d8e-778f-4eea-a94b-fe231b0922d2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc\" (UID: \"50f91d8e-778f-4eea-a94b-fe231b0922d2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc"
Jul 30 02:34:07.243748 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:07.243613    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50f91d8e-778f-4eea-a94b-fe231b0922d2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc\" (UID: \"50f91d8e-778f-4eea-a94b-fe231b0922d2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc"
Jul 30 02:34:07.244017 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:07.243996    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50f91d8e-778f-4eea-a94b-fe231b0922d2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc\" (UID: \"50f91d8e-778f-4eea-a94b-fe231b0922d2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc"
Jul 30 02:34:07.244081 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:07.244012    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50f91d8e-778f-4eea-a94b-fe231b0922d2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc\" (UID: \"50f91d8e-778f-4eea-a94b-fe231b0922d2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc"
Jul 30 02:34:07.251758 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:07.251732    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr9px\" (UniqueName: \"kubernetes.io/projected/50f91d8e-778f-4eea-a94b-fe231b0922d2-kube-api-access-zr9px\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc\" (UID: \"50f91d8e-778f-4eea-a94b-fe231b0922d2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc"
Jul 30 02:34:07.360883 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:07.360853    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc"
Jul 30 02:34:07.502508 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:07.502479    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc"]
Jul 30 02:34:07.504759 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:34:07.504733    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f91d8e_778f_4eea_a94b_fe231b0922d2.slice/crio-0b4712538a6569c93e43efeb113ec6397c76517fd968371af999e9a385867e9c WatchSource:0}: Error finding container 0b4712538a6569c93e43efeb113ec6397c76517fd968371af999e9a385867e9c: Status 404 returned error can't find the container with id 0b4712538a6569c93e43efeb113ec6397c76517fd968371af999e9a385867e9c
Jul 30 02:34:08.450027 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:08.449996    2565 generic.go:356] "Generic (PLEG): container finished" podID="50f91d8e-778f-4eea-a94b-fe231b0922d2" containerID="445e7de8701a50ae43d2f81a2e74a966f101f20bc2a035a4a3b940cb540dc4ae" exitCode=0
Jul 30 02:34:08.450386 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:08.450062    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc" event={"ID":"50f91d8e-778f-4eea-a94b-fe231b0922d2","Type":"ContainerDied","Data":"445e7de8701a50ae43d2f81a2e74a966f101f20bc2a035a4a3b940cb540dc4ae"}
Jul 30 02:34:08.450386 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:08.450099    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc" event={"ID":"50f91d8e-778f-4eea-a94b-fe231b0922d2","Type":"ContainerStarted","Data":"0b4712538a6569c93e43efeb113ec6397c76517fd968371af999e9a385867e9c"}
Jul 30 02:34:09.457249 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:09.457142    2565 generic.go:356] "Generic (PLEG): container finished" podID="50f91d8e-778f-4eea-a94b-fe231b0922d2" containerID="177074cd15cea36832ca24f44971645e9ea3e9ef1b0ced96ae45821de3c0bcf6" exitCode=0
Jul 30 02:34:09.457249 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:09.457203    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc" event={"ID":"50f91d8e-778f-4eea-a94b-fe231b0922d2","Type":"ContainerDied","Data":"177074cd15cea36832ca24f44971645e9ea3e9ef1b0ced96ae45821de3c0bcf6"}
Jul 30 02:34:10.462872 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:10.462840    2565 generic.go:356] "Generic (PLEG): container finished" podID="50f91d8e-778f-4eea-a94b-fe231b0922d2" containerID="28beba4a2524827fffa6c656d04c59e238250f3fd9d0b9971ffabb5787b090b0" exitCode=0
Jul 30 02:34:10.463263 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:10.462933    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc" event={"ID":"50f91d8e-778f-4eea-a94b-fe231b0922d2","Type":"ContainerDied","Data":"28beba4a2524827fffa6c656d04c59e238250f3fd9d0b9971ffabb5787b090b0"}
Jul 30 02:34:11.593061 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:11.593038    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc"
Jul 30 02:34:11.678851 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:11.678824    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr9px\" (UniqueName: \"kubernetes.io/projected/50f91d8e-778f-4eea-a94b-fe231b0922d2-kube-api-access-zr9px\") pod \"50f91d8e-778f-4eea-a94b-fe231b0922d2\" (UID: \"50f91d8e-778f-4eea-a94b-fe231b0922d2\") "
Jul 30 02:34:11.678989 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:11.678874    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50f91d8e-778f-4eea-a94b-fe231b0922d2-bundle\") pod \"50f91d8e-778f-4eea-a94b-fe231b0922d2\" (UID: \"50f91d8e-778f-4eea-a94b-fe231b0922d2\") "
Jul 30 02:34:11.678989 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:11.678914    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50f91d8e-778f-4eea-a94b-fe231b0922d2-util\") pod \"50f91d8e-778f-4eea-a94b-fe231b0922d2\" (UID: \"50f91d8e-778f-4eea-a94b-fe231b0922d2\") "
Jul 30 02:34:11.679785 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:11.679759    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f91d8e-778f-4eea-a94b-fe231b0922d2-bundle" (OuterVolumeSpecName: "bundle") pod "50f91d8e-778f-4eea-a94b-fe231b0922d2" (UID: "50f91d8e-778f-4eea-a94b-fe231b0922d2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:34:11.680947 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:11.680921    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f91d8e-778f-4eea-a94b-fe231b0922d2-kube-api-access-zr9px" (OuterVolumeSpecName: "kube-api-access-zr9px") pod "50f91d8e-778f-4eea-a94b-fe231b0922d2" (UID: "50f91d8e-778f-4eea-a94b-fe231b0922d2"). InnerVolumeSpecName "kube-api-access-zr9px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:34:11.684856 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:11.684812    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f91d8e-778f-4eea-a94b-fe231b0922d2-util" (OuterVolumeSpecName: "util") pod "50f91d8e-778f-4eea-a94b-fe231b0922d2" (UID: "50f91d8e-778f-4eea-a94b-fe231b0922d2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:34:11.780073 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:11.779984    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr9px\" (UniqueName: \"kubernetes.io/projected/50f91d8e-778f-4eea-a94b-fe231b0922d2-kube-api-access-zr9px\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:34:11.780073 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:11.780014    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50f91d8e-778f-4eea-a94b-fe231b0922d2-bundle\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:34:11.780073 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:11.780026    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50f91d8e-778f-4eea-a94b-fe231b0922d2-util\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:34:12.471838 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:12.471808    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc"
Jul 30 02:34:12.472005 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:12.471804    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmshqc" event={"ID":"50f91d8e-778f-4eea-a94b-fe231b0922d2","Type":"ContainerDied","Data":"0b4712538a6569c93e43efeb113ec6397c76517fd968371af999e9a385867e9c"}
Jul 30 02:34:12.472005 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:12.471937    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b4712538a6569c93e43efeb113ec6397c76517fd968371af999e9a385867e9c"
Jul 30 02:34:24.257058 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.257027    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"]
Jul 30 02:34:24.257483 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.257373    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f91d8e-778f-4eea-a94b-fe231b0922d2" containerName="extract"
Jul 30 02:34:24.257483 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.257386    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f91d8e-778f-4eea-a94b-fe231b0922d2" containerName="extract"
Jul 30 02:34:24.257483 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.257398    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f91d8e-778f-4eea-a94b-fe231b0922d2" containerName="pull"
Jul 30 02:34:24.257483 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.257406    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f91d8e-778f-4eea-a94b-fe231b0922d2" containerName="pull"
Jul 30 02:34:24.257483 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.257426    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f91d8e-778f-4eea-a94b-fe231b0922d2" containerName="util"
Jul 30 02:34:24.257483 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.257432    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f91d8e-778f-4eea-a94b-fe231b0922d2" containerName="util"
Jul 30 02:34:24.257483 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.257479    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f91d8e-778f-4eea-a94b-fe231b0922d2" containerName="extract"
Jul 30 02:34:24.259944 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.259926    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"
Jul 30 02:34:24.262860 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.262835    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 30 02:34:24.263152 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.263127    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 30 02:34:24.263274 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.263214    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 30 02:34:24.263274 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.263232    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 30 02:34:24.263274 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.263232    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fv95l\""
Jul 30 02:34:24.278056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.278033    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"]
Jul 30 02:34:24.376063 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.376028    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/93cde41a-963f-4f29-814d-4c909d54efa3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-684d7c5d99-sc8pn\" (UID: \"93cde41a-963f-4f29-814d-4c909d54efa3\") " pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"
Jul 30 02:34:24.376260 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.376083    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wftb\" (UniqueName: \"kubernetes.io/projected/93cde41a-963f-4f29-814d-4c909d54efa3-kube-api-access-6wftb\") pod \"opendatahub-operator-controller-manager-684d7c5d99-sc8pn\" (UID: \"93cde41a-963f-4f29-814d-4c909d54efa3\") " pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"
Jul 30 02:34:24.376260 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.376147    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/93cde41a-963f-4f29-814d-4c909d54efa3-manifests\") pod \"opendatahub-operator-controller-manager-684d7c5d99-sc8pn\" (UID: \"93cde41a-963f-4f29-814d-4c909d54efa3\") " pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"
Jul 30 02:34:24.376260 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.376234    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93cde41a-963f-4f29-814d-4c909d54efa3-webhook-cert\") pod \"opendatahub-operator-controller-manager-684d7c5d99-sc8pn\" (UID: \"93cde41a-963f-4f29-814d-4c909d54efa3\") " pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"
Jul 30 02:34:24.376375 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.376283    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93cde41a-963f-4f29-814d-4c909d54efa3-tmp\") pod \"opendatahub-operator-controller-manager-684d7c5d99-sc8pn\" (UID: \"93cde41a-963f-4f29-814d-4c909d54efa3\") " pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"
Jul 30 02:34:24.477563 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.477523    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wftb\" (UniqueName: \"kubernetes.io/projected/93cde41a-963f-4f29-814d-4c909d54efa3-kube-api-access-6wftb\") pod \"opendatahub-operator-controller-manager-684d7c5d99-sc8pn\" (UID: \"93cde41a-963f-4f29-814d-4c909d54efa3\") " pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"
Jul 30 02:34:24.477563 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.477567    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/93cde41a-963f-4f29-814d-4c909d54efa3-manifests\") pod \"opendatahub-operator-controller-manager-684d7c5d99-sc8pn\" (UID: \"93cde41a-963f-4f29-814d-4c909d54efa3\") " pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"
Jul 30 02:34:24.477794 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.477589    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93cde41a-963f-4f29-814d-4c909d54efa3-webhook-cert\") pod \"opendatahub-operator-controller-manager-684d7c5d99-sc8pn\" (UID: \"93cde41a-963f-4f29-814d-4c909d54efa3\") " pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"
Jul 30 02:34:24.477794 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.477621    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93cde41a-963f-4f29-814d-4c909d54efa3-tmp\") pod \"opendatahub-operator-controller-manager-684d7c5d99-sc8pn\" (UID: \"93cde41a-963f-4f29-814d-4c909d54efa3\") " pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"
Jul 30 02:34:24.477794 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.477641    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/93cde41a-963f-4f29-814d-4c909d54efa3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-684d7c5d99-sc8pn\" (UID: \"93cde41a-963f-4f29-814d-4c909d54efa3\") " pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"
Jul 30 02:34:24.478053 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.478030    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/93cde41a-963f-4f29-814d-4c909d54efa3-manifests\") pod \"opendatahub-operator-controller-manager-684d7c5d99-sc8pn\" (UID: \"93cde41a-963f-4f29-814d-4c909d54efa3\") " pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"
Jul 30 02:34:24.480067 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.480033    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93cde41a-963f-4f29-814d-4c909d54efa3-tmp\") pod \"opendatahub-operator-controller-manager-684d7c5d99-sc8pn\" (UID: \"93cde41a-963f-4f29-814d-4c909d54efa3\") " pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"
Jul 30 02:34:24.480195 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.480110    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/93cde41a-963f-4f29-814d-4c909d54efa3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-684d7c5d99-sc8pn\" (UID: \"93cde41a-963f-4f29-814d-4c909d54efa3\") " pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"
Jul 30 02:34:24.480195 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.480150    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93cde41a-963f-4f29-814d-4c909d54efa3-webhook-cert\") pod \"opendatahub-operator-controller-manager-684d7c5d99-sc8pn\" (UID: \"93cde41a-963f-4f29-814d-4c909d54efa3\") " pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"
Jul 30 02:34:24.485569 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.485552    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wftb\" (UniqueName: \"kubernetes.io/projected/93cde41a-963f-4f29-814d-4c909d54efa3-kube-api-access-6wftb\") pod \"opendatahub-operator-controller-manager-684d7c5d99-sc8pn\" (UID: \"93cde41a-963f-4f29-814d-4c909d54efa3\") " pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"
Jul 30 02:34:24.570176 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.570143    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"
Jul 30 02:34:24.724477 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:24.724389    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"]
Jul 30 02:34:24.726863 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:34:24.726833    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93cde41a_963f_4f29_814d_4c909d54efa3.slice/crio-1ed61cc7e2e020deba6162a3cf045d5bfaf72632c2a22680805c3d9edaf48b15 WatchSource:0}: Error finding container 1ed61cc7e2e020deba6162a3cf045d5bfaf72632c2a22680805c3d9edaf48b15: Status 404 returned error can't find the container with id 1ed61cc7e2e020deba6162a3cf045d5bfaf72632c2a22680805c3d9edaf48b15
Jul 30 02:34:25.527764 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:25.527725    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn" event={"ID":"93cde41a-963f-4f29-814d-4c909d54efa3","Type":"ContainerStarted","Data":"1ed61cc7e2e020deba6162a3cf045d5bfaf72632c2a22680805c3d9edaf48b15"}
Jul 30 02:34:28.541842 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:28.541811    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn" event={"ID":"93cde41a-963f-4f29-814d-4c909d54efa3","Type":"ContainerStarted","Data":"0fe407946fffcda8cd4219eceaf04b93a23384f787034a361b994ac7670c52b8"}
Jul 30 02:34:29.547728 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:29.547693    2565 generic.go:356] "Generic (PLEG): container finished" podID="93cde41a-963f-4f29-814d-4c909d54efa3" containerID="0fe407946fffcda8cd4219eceaf04b93a23384f787034a361b994ac7670c52b8" exitCode=0
Jul 30 02:34:29.548106 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:29.547741    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn" event={"ID":"93cde41a-963f-4f29-814d-4c909d54efa3","Type":"ContainerDied","Data":"0fe407946fffcda8cd4219eceaf04b93a23384f787034a361b994ac7670c52b8"}
Jul 30 02:34:30.554236 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:30.554203    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn" event={"ID":"93cde41a-963f-4f29-814d-4c909d54efa3","Type":"ContainerStarted","Data":"12e829bf7ade26ab0f49fdb3e06f7ff327eb3865a8297b52d2d42aee7f3471c3"}
Jul 30 02:34:30.554579 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:30.554331    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"
Jul 30 02:34:30.593453 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:30.593405    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn" podStartSLOduration=2.208474773 podStartE2EDuration="6.593391166s" totalImagesPullingTime="3.316352068s" totalInitContainerRuntime="1.068564325s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:34:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:34:30.591632476 +0000 UTC m=+506.238415884" watchObservedRunningTime="2026-07-30 02:34:30.593391166 +0000 UTC m=+506.240174571"
Jul 30 02:34:41.560938 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:41.560905    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-684d7c5d99-sc8pn"
Jul 30 02:34:46.688335 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:46.688304    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5b44668b45-jz27k"]
Jul 30 02:34:46.696335 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:46.696310    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b44668b45-jz27k"
Jul 30 02:34:46.699444 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:46.699418    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 30 02:34:46.699666 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:46.699456    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 30 02:34:46.699833 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:46.699615    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 30 02:34:46.699932 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:46.699633    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zl6b8\""
Jul 30 02:34:46.699995 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:46.699685    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 30 02:34:46.700590 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:46.700569    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b44668b45-jz27k"]
Jul 30 02:34:46.861727 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:46.861696    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fe6e745-14ba-4e8b-b8b0-72be885b4169-tmp\") pod \"kube-auth-proxy-5b44668b45-jz27k\" (UID: \"8fe6e745-14ba-4e8b-b8b0-72be885b4169\") " pod="openshift-ingress/kube-auth-proxy-5b44668b45-jz27k"
Jul 30 02:34:46.861727 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:46.861732    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th65m\" (UniqueName: \"kubernetes.io/projected/8fe6e745-14ba-4e8b-b8b0-72be885b4169-kube-api-access-th65m\") pod \"kube-auth-proxy-5b44668b45-jz27k\" (UID: \"8fe6e745-14ba-4e8b-b8b0-72be885b4169\") " pod="openshift-ingress/kube-auth-proxy-5b44668b45-jz27k"
Jul 30 02:34:46.861917 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:46.861776    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fe6e745-14ba-4e8b-b8b0-72be885b4169-tls-certs\") pod \"kube-auth-proxy-5b44668b45-jz27k\" (UID: \"8fe6e745-14ba-4e8b-b8b0-72be885b4169\") " pod="openshift-ingress/kube-auth-proxy-5b44668b45-jz27k"
Jul 30 02:34:46.962608 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:46.962529    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fe6e745-14ba-4e8b-b8b0-72be885b4169-tmp\") pod \"kube-auth-proxy-5b44668b45-jz27k\" (UID: \"8fe6e745-14ba-4e8b-b8b0-72be885b4169\") " pod="openshift-ingress/kube-auth-proxy-5b44668b45-jz27k"
Jul 30 02:34:46.962608 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:46.962567    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th65m\" (UniqueName: \"kubernetes.io/projected/8fe6e745-14ba-4e8b-b8b0-72be885b4169-kube-api-access-th65m\") pod \"kube-auth-proxy-5b44668b45-jz27k\" (UID: \"8fe6e745-14ba-4e8b-b8b0-72be885b4169\") " pod="openshift-ingress/kube-auth-proxy-5b44668b45-jz27k"
Jul 30 02:34:46.962776 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:46.962608    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fe6e745-14ba-4e8b-b8b0-72be885b4169-tls-certs\") pod \"kube-auth-proxy-5b44668b45-jz27k\" (UID: \"8fe6e745-14ba-4e8b-b8b0-72be885b4169\") " pod="openshift-ingress/kube-auth-proxy-5b44668b45-jz27k"
Jul 30 02:34:46.964870 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:46.964839    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fe6e745-14ba-4e8b-b8b0-72be885b4169-tmp\") pod \"kube-auth-proxy-5b44668b45-jz27k\" (UID: \"8fe6e745-14ba-4e8b-b8b0-72be885b4169\") " pod="openshift-ingress/kube-auth-proxy-5b44668b45-jz27k"
Jul 30 02:34:46.965136 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:46.965119    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fe6e745-14ba-4e8b-b8b0-72be885b4169-tls-certs\") pod \"kube-auth-proxy-5b44668b45-jz27k\" (UID: \"8fe6e745-14ba-4e8b-b8b0-72be885b4169\") " pod="openshift-ingress/kube-auth-proxy-5b44668b45-jz27k"
Jul 30 02:34:46.976576 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:46.976553    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th65m\" (UniqueName: \"kubernetes.io/projected/8fe6e745-14ba-4e8b-b8b0-72be885b4169-kube-api-access-th65m\") pod \"kube-auth-proxy-5b44668b45-jz27k\" (UID: \"8fe6e745-14ba-4e8b-b8b0-72be885b4169\") " pod="openshift-ingress/kube-auth-proxy-5b44668b45-jz27k"
Jul 30 02:34:47.008227 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.008202    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b44668b45-jz27k"
Jul 30 02:34:47.142673 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.142647    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b44668b45-jz27k"]
Jul 30 02:34:47.144063 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:34:47.144036    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe6e745_14ba_4e8b_b8b0_72be885b4169.slice/crio-468aadd0f28fbc7e92d7a02cfd0247ec8758803fe1f6d78a2b0d7f32cebd2c04 WatchSource:0}: Error finding container 468aadd0f28fbc7e92d7a02cfd0247ec8758803fe1f6d78a2b0d7f32cebd2c04: Status 404 returned error can't find the container with id 468aadd0f28fbc7e92d7a02cfd0247ec8758803fe1f6d78a2b0d7f32cebd2c04
Jul 30 02:34:47.582034 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.582001    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"]
Jul 30 02:34:47.587031 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.587010    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.589895 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.589865    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 30 02:34:47.589895 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.589873    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 30 02:34:47.589895 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.589893    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 30 02:34:47.590142 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.589865    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 30 02:34:47.590142 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.589924    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qjgdm\""
Jul 30 02:34:47.604238 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.604208    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"]
Jul 30 02:34:47.626662 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.626627    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b44668b45-jz27k" event={"ID":"8fe6e745-14ba-4e8b-b8b0-72be885b4169","Type":"ContainerStarted","Data":"468aadd0f28fbc7e92d7a02cfd0247ec8758803fe1f6d78a2b0d7f32cebd2c04"}
Jul 30 02:34:47.769769 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.769730    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f9jb\" (UniqueName: \"kubernetes.io/projected/db9e89e9-2ea1-42dc-907a-e600187c2eb6-kube-api-access-2f9jb\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.770236 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.769778    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.770236 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.769826    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.770236 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.769867    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.770236 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.769904    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9e89e9-2ea1-42dc-907a-e600187c2eb6-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.770236 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.769928    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/db9e89e9-2ea1-42dc-907a-e600187c2eb6-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.770236 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.769957    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.871048 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.870973    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.871048 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.871020    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9e89e9-2ea1-42dc-907a-e600187c2eb6-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.871048 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.871046    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/db9e89e9-2ea1-42dc-907a-e600187c2eb6-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.871331 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.871071    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.871331 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.871124    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9jb\" (UniqueName: \"kubernetes.io/projected/db9e89e9-2ea1-42dc-907a-e600187c2eb6-kube-api-access-2f9jb\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.871331 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.871151    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.871881 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.871478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.872242 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.872190    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.874051 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.874015    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9e89e9-2ea1-42dc-907a-e600187c2eb6-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.874051 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.874029    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.874239 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.874200    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/db9e89e9-2ea1-42dc-907a-e600187c2eb6-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.874525 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.874499    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.879343 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.879320    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f9jb\" (UniqueName: \"kubernetes.io/projected/db9e89e9-2ea1-42dc-907a-e600187c2eb6-kube-api-access-2f9jb\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.879543 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.879524    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-7jmbj\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:47.902625 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:47.902599    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:48.054877 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:48.054848    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"]
Jul 30 02:34:48.057217 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:34:48.057185    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb9e89e9_2ea1_42dc_907a_e600187c2eb6.slice/crio-fcfe4043e76920564b5be1e1f062a6fcbb6e6569565d2f5cd2b47c8ea1eaa123 WatchSource:0}: Error finding container fcfe4043e76920564b5be1e1f062a6fcbb6e6569565d2f5cd2b47c8ea1eaa123: Status 404 returned error can't find the container with id fcfe4043e76920564b5be1e1f062a6fcbb6e6569565d2f5cd2b47c8ea1eaa123
Jul 30 02:34:48.632797 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:48.632750    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj" event={"ID":"db9e89e9-2ea1-42dc-907a-e600187c2eb6","Type":"ContainerStarted","Data":"fcfe4043e76920564b5be1e1f062a6fcbb6e6569565d2f5cd2b47c8ea1eaa123"}
Jul 30 02:34:50.644244 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:50.644208    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b44668b45-jz27k" event={"ID":"8fe6e745-14ba-4e8b-b8b0-72be885b4169","Type":"ContainerStarted","Data":"09550422579f36aa9d159c13860ec5fc886d2f55ad54abdd9a411c83718a5e97"}
Jul 30 02:34:50.661207 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:50.661012    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5b44668b45-jz27k" podStartSLOduration=2.230365705 podStartE2EDuration="4.660993435s" totalImagesPullingTime="2.43062773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:34:50.660475452 +0000 UTC m=+526.307258858" watchObservedRunningTime="2026-07-30 02:34:50.660993435 +0000 UTC m=+526.307776843"
Jul 30 02:34:51.444013 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:51.443975    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:34:51.444115 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:51.444041    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:34:51.650436 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:51.650386    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj" event={"ID":"db9e89e9-2ea1-42dc-907a-e600187c2eb6","Type":"ContainerStarted","Data":"bb48458e93439a56bec32d19af673f1278a9aa3895729bd0d4e417c9c9470ed4"}
Jul 30 02:34:51.650884 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:51.650648    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:51.682812 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:51.682770    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj" podStartSLOduration=1.2980964400000001 podStartE2EDuration="4.682754201s" totalImagesPullingTime="3.384657761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:34:51.680527654 +0000 UTC m=+527.327311057" watchObservedRunningTime="2026-07-30 02:34:51.682754201 +0000 UTC m=+527.329537605"
Jul 30 02:34:52.656599 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:52.656569    2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-7jmbj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 30 02:34:52.656962 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:52.656615    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj" podUID="db9e89e9-2ea1-42dc-907a-e600187c2eb6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 02:34:53.289562 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.289532    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"]
Jul 30 02:34:53.292934 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.292909    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.296213 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.296187    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-8bl68\""
Jul 30 02:34:53.313312 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.313290    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"]
Jul 30 02:34:53.325530 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.325508    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3ad8b3-6927-4c20-a313-51510a35bf13-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.325663 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.325539    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3ad8b3-6927-4c20-a313-51510a35bf13-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.325663 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.325577    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3ad8b3-6927-4c20-a313-51510a35bf13-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.325663 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.325619    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3ad8b3-6927-4c20-a313-51510a35bf13-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.325663 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.325658    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c3ad8b3-6927-4c20-a313-51510a35bf13-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.325875 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.325683    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3ad8b3-6927-4c20-a313-51510a35bf13-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.325875 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.325767    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3ad8b3-6927-4c20-a313-51510a35bf13-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.325875 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.325796    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3ad8b3-6927-4c20-a313-51510a35bf13-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.325875 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.325823    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slpfv\" (UniqueName: \"kubernetes.io/projected/0c3ad8b3-6927-4c20-a313-51510a35bf13-kube-api-access-slpfv\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.426867 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.426823    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3ad8b3-6927-4c20-a313-51510a35bf13-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.427047 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.426906    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3ad8b3-6927-4c20-a313-51510a35bf13-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.427047 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.426955    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c3ad8b3-6927-4c20-a313-51510a35bf13-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.427047 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.426988    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3ad8b3-6927-4c20-a313-51510a35bf13-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.427232 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.427049    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3ad8b3-6927-4c20-a313-51510a35bf13-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.427232 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.427080    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3ad8b3-6927-4c20-a313-51510a35bf13-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.427232 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.427112    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slpfv\" (UniqueName: \"kubernetes.io/projected/0c3ad8b3-6927-4c20-a313-51510a35bf13-kube-api-access-slpfv\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.427232 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.427150    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3ad8b3-6927-4c20-a313-51510a35bf13-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.427232 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.427200    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3ad8b3-6927-4c20-a313-51510a35bf13-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.427597 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.427566    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3ad8b3-6927-4c20-a313-51510a35bf13-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.427829 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.427779    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3ad8b3-6927-4c20-a313-51510a35bf13-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.427973 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.427852    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3ad8b3-6927-4c20-a313-51510a35bf13-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.428062 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.427899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3ad8b3-6927-4c20-a313-51510a35bf13-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.428127 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.428109    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3ad8b3-6927-4c20-a313-51510a35bf13-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.429549 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.429526    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3ad8b3-6927-4c20-a313-51510a35bf13-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.429909 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.429890    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3ad8b3-6927-4c20-a313-51510a35bf13-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.443527 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.442322    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slpfv\" (UniqueName: \"kubernetes.io/projected/0c3ad8b3-6927-4c20-a313-51510a35bf13-kube-api-access-slpfv\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.443527 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.442691    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c3ad8b3-6927-4c20-a313-51510a35bf13-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl\" (UID: \"0c3ad8b3-6927-4c20-a313-51510a35bf13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.603113 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.603008    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:34:53.747301 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:53.747277    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"]
Jul 30 02:34:53.748807 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:34:53.748781    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3ad8b3_6927_4c20_a313_51510a35bf13.slice/crio-9e8c976bb474cf196a6c616bdabd738e79cfecd3c15e40115bccb8bd6335aaa1 WatchSource:0}: Error finding container 9e8c976bb474cf196a6c616bdabd738e79cfecd3c15e40115bccb8bd6335aaa1: Status 404 returned error can't find the container with id 9e8c976bb474cf196a6c616bdabd738e79cfecd3c15e40115bccb8bd6335aaa1
Jul 30 02:34:54.666556 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.666502    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl" event={"ID":"0c3ad8b3-6927-4c20-a313-51510a35bf13","Type":"ContainerStarted","Data":"9e8c976bb474cf196a6c616bdabd738e79cfecd3c15e40115bccb8bd6335aaa1"}
Jul 30 02:34:54.785933 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.785895    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5kntq"]
Jul 30 02:34:54.790485 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.790460    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5kntq"
Jul 30 02:34:54.793296 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.793276    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 30 02:34:54.793404 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.793303    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-7slqs\""
Jul 30 02:34:54.805733 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.805712    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-x28m9"]
Jul 30 02:34:54.808859 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.808839    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5kntq"]
Jul 30 02:34:54.808958 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.808945    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x28m9"
Jul 30 02:34:54.811855 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.811836    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-wn596\""
Jul 30 02:34:54.812149 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.812127    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 30 02:34:54.818533 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.818509    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-x28m9"]
Jul 30 02:34:54.839237 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.839208    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf4f1044-78f4-4086-aac0-406665379140-cert\") pod \"kserve-controller-manager-d6f4b65b8-5kntq\" (UID: \"cf4f1044-78f4-4086-aac0-406665379140\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5kntq"
Jul 30 02:34:54.839356 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.839246    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9pcd\" (UniqueName: \"kubernetes.io/projected/cf4f1044-78f4-4086-aac0-406665379140-kube-api-access-h9pcd\") pod \"kserve-controller-manager-d6f4b65b8-5kntq\" (UID: \"cf4f1044-78f4-4086-aac0-406665379140\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5kntq"
Jul 30 02:34:54.839356 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.839315    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e9e9962-00bd-4256-a254-276c940b6d80-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-x28m9\" (UID: \"6e9e9962-00bd-4256-a254-276c940b6d80\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x28m9"
Jul 30 02:34:54.839446 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.839379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4xpm\" (UniqueName: \"kubernetes.io/projected/6e9e9962-00bd-4256-a254-276c940b6d80-kube-api-access-n4xpm\") pod \"llmisvc-controller-manager-777c6bd7d5-x28m9\" (UID: \"6e9e9962-00bd-4256-a254-276c940b6d80\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x28m9"
Jul 30 02:34:54.940043 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.939967    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf4f1044-78f4-4086-aac0-406665379140-cert\") pod \"kserve-controller-manager-d6f4b65b8-5kntq\" (UID: \"cf4f1044-78f4-4086-aac0-406665379140\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5kntq"
Jul 30 02:34:54.940043 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.940005    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9pcd\" (UniqueName: \"kubernetes.io/projected/cf4f1044-78f4-4086-aac0-406665379140-kube-api-access-h9pcd\") pod \"kserve-controller-manager-d6f4b65b8-5kntq\" (UID: \"cf4f1044-78f4-4086-aac0-406665379140\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5kntq"
Jul 30 02:34:54.940295 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.940052    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e9e9962-00bd-4256-a254-276c940b6d80-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-x28m9\" (UID: \"6e9e9962-00bd-4256-a254-276c940b6d80\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x28m9"
Jul 30 02:34:54.940295 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.940096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4xpm\" (UniqueName: \"kubernetes.io/projected/6e9e9962-00bd-4256-a254-276c940b6d80-kube-api-access-n4xpm\") pod \"llmisvc-controller-manager-777c6bd7d5-x28m9\" (UID: \"6e9e9962-00bd-4256-a254-276c940b6d80\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x28m9"
Jul 30 02:34:54.940295 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:34:54.940129    2565 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 30 02:34:54.940295 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:34:54.940223    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf4f1044-78f4-4086-aac0-406665379140-cert podName:cf4f1044-78f4-4086-aac0-406665379140 nodeName:}" failed. No retries permitted until 2026-07-30 02:34:55.440201095 +0000 UTC m=+531.086984480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/cf4f1044-78f4-4086-aac0-406665379140-cert") pod "kserve-controller-manager-d6f4b65b8-5kntq" (UID: "cf4f1044-78f4-4086-aac0-406665379140") : secret "kserve-webhook-server-cert" not found
Jul 30 02:34:54.942620 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.942602    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e9e9962-00bd-4256-a254-276c940b6d80-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-x28m9\" (UID: \"6e9e9962-00bd-4256-a254-276c940b6d80\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x28m9"
Jul 30 02:34:54.953143 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.953119    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4xpm\" (UniqueName: \"kubernetes.io/projected/6e9e9962-00bd-4256-a254-276c940b6d80-kube-api-access-n4xpm\") pod \"llmisvc-controller-manager-777c6bd7d5-x28m9\" (UID: \"6e9e9962-00bd-4256-a254-276c940b6d80\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x28m9"
Jul 30 02:34:54.961472 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:54.961451    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9pcd\" (UniqueName: \"kubernetes.io/projected/cf4f1044-78f4-4086-aac0-406665379140-kube-api-access-h9pcd\") pod \"kserve-controller-manager-d6f4b65b8-5kntq\" (UID: \"cf4f1044-78f4-4086-aac0-406665379140\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5kntq"
Jul 30 02:34:55.120147 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:55.120107    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x28m9"
Jul 30 02:34:55.249679 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:55.249601    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-x28m9"]
Jul 30 02:34:55.251748 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:34:55.251719    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6e9e9962_00bd_4256_a254_276c940b6d80.slice/crio-4b2bbc3ef811b4f00ae4cd0a1c239d9fb62db1cc264e89328158dd2d0fa0bfb7 WatchSource:0}: Error finding container 4b2bbc3ef811b4f00ae4cd0a1c239d9fb62db1cc264e89328158dd2d0fa0bfb7: Status 404 returned error can't find the container with id 4b2bbc3ef811b4f00ae4cd0a1c239d9fb62db1cc264e89328158dd2d0fa0bfb7
Jul 30 02:34:55.443785 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:55.443755    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf4f1044-78f4-4086-aac0-406665379140-cert\") pod \"kserve-controller-manager-d6f4b65b8-5kntq\" (UID: \"cf4f1044-78f4-4086-aac0-406665379140\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5kntq"
Jul 30 02:34:55.446276 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:55.446254    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf4f1044-78f4-4086-aac0-406665379140-cert\") pod \"kserve-controller-manager-d6f4b65b8-5kntq\" (UID: \"cf4f1044-78f4-4086-aac0-406665379140\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5kntq"
Jul 30 02:34:55.656232 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:55.656146    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:55.671814 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:55.671788    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x28m9" event={"ID":"6e9e9962-00bd-4256-a254-276c940b6d80","Type":"ContainerStarted","Data":"4b2bbc3ef811b4f00ae4cd0a1c239d9fb62db1cc264e89328158dd2d0fa0bfb7"}
Jul 30 02:34:55.701265 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:55.701241    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5kntq"
Jul 30 02:34:55.843849 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:55.843816    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5kntq"]
Jul 30 02:34:55.844777 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:34:55.844752    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4f1044_78f4_4086_aac0_406665379140.slice/crio-d62cf63a419d3e38d64f4fd98d4c0c0064707a96ce505d7b98daca4b7afc0ddd WatchSource:0}: Error finding container d62cf63a419d3e38d64f4fd98d4c0c0064707a96ce505d7b98daca4b7afc0ddd: Status 404 returned error can't find the container with id d62cf63a419d3e38d64f4fd98d4c0c0064707a96ce505d7b98daca4b7afc0ddd
Jul 30 02:34:56.680893 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:56.680840    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5kntq" event={"ID":"cf4f1044-78f4-4086-aac0-406665379140","Type":"ContainerStarted","Data":"d62cf63a419d3e38d64f4fd98d4c0c0064707a96ce505d7b98daca4b7afc0ddd"}
Jul 30 02:34:57.782812 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:57.782774    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"]
Jul 30 02:34:57.783316 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:57.783002    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj" podUID="db9e89e9-2ea1-42dc-907a-e600187c2eb6" containerName="discovery" containerID="cri-o://bb48458e93439a56bec32d19af673f1278a9aa3895729bd0d4e417c9c9470ed4" gracePeriod=30
Jul 30 02:34:58.535315 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.535287    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:58.577452 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.577427    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-csr-dns-cert\") pod \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") "
Jul 30 02:34:58.577558 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.577481    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f9jb\" (UniqueName: \"kubernetes.io/projected/db9e89e9-2ea1-42dc-907a-e600187c2eb6-kube-api-access-2f9jb\") pod \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") "
Jul 30 02:34:58.577558 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.577508    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/db9e89e9-2ea1-42dc-907a-e600187c2eb6-cacerts\") pod \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") "
Jul 30 02:34:58.577667 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.577559    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9e89e9-2ea1-42dc-907a-e600187c2eb6-local-certs\") pod \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") "
Jul 30 02:34:58.577667 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.577597    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-token\") pod \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") "
Jul 30 02:34:58.577667 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.577630    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-csr-ca-configmap\") pod \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") "
Jul 30 02:34:58.577808 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.577700    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-kubeconfig\") pod \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\" (UID: \"db9e89e9-2ea1-42dc-907a-e600187c2eb6\") "
Jul 30 02:34:58.578313 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.578268    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "db9e89e9-2ea1-42dc-907a-e600187c2eb6" (UID: "db9e89e9-2ea1-42dc-907a-e600187c2eb6"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 02:34:58.580261 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.580232    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "db9e89e9-2ea1-42dc-907a-e600187c2eb6" (UID: "db9e89e9-2ea1-42dc-907a-e600187c2eb6"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 02:34:58.580353 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.580253    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "db9e89e9-2ea1-42dc-907a-e600187c2eb6" (UID: "db9e89e9-2ea1-42dc-907a-e600187c2eb6"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 02:34:58.580353 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.580228    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9e89e9-2ea1-42dc-907a-e600187c2eb6-kube-api-access-2f9jb" (OuterVolumeSpecName: "kube-api-access-2f9jb") pod "db9e89e9-2ea1-42dc-907a-e600187c2eb6" (UID: "db9e89e9-2ea1-42dc-907a-e600187c2eb6"). InnerVolumeSpecName "kube-api-access-2f9jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:34:58.580353 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.580298    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9e89e9-2ea1-42dc-907a-e600187c2eb6-local-certs" (OuterVolumeSpecName: "local-certs") pod "db9e89e9-2ea1-42dc-907a-e600187c2eb6" (UID: "db9e89e9-2ea1-42dc-907a-e600187c2eb6"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:34:58.580353 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.580326    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db9e89e9-2ea1-42dc-907a-e600187c2eb6-cacerts" (OuterVolumeSpecName: "cacerts") pod "db9e89e9-2ea1-42dc-907a-e600187c2eb6" (UID: "db9e89e9-2ea1-42dc-907a-e600187c2eb6"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 02:34:58.580499 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.580479    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-token" (OuterVolumeSpecName: "istio-token") pod "db9e89e9-2ea1-42dc-907a-e600187c2eb6" (UID: "db9e89e9-2ea1-42dc-907a-e600187c2eb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:34:58.678531 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.678502    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f9jb\" (UniqueName: \"kubernetes.io/projected/db9e89e9-2ea1-42dc-907a-e600187c2eb6-kube-api-access-2f9jb\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:34:58.678531 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.678527    2565 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/db9e89e9-2ea1-42dc-907a-e600187c2eb6-cacerts\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:34:58.678680 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.678537    2565 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9e89e9-2ea1-42dc-907a-e600187c2eb6-local-certs\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:34:58.678680 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.678547    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-token\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:34:58.678680 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.678579    2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-csr-ca-configmap\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:34:58.678680 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.678587    2565 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-kubeconfig\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:34:58.678680 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.678596    2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/db9e89e9-2ea1-42dc-907a-e600187c2eb6-istio-csr-dns-cert\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:34:58.692220 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.692148    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x28m9" event={"ID":"6e9e9962-00bd-4256-a254-276c940b6d80","Type":"ContainerStarted","Data":"3583359306e9ddaccd2a38395c452e91f09f3320df9993640d4b3a0942cfab68"}
Jul 30 02:34:58.693799 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.693781    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5kntq"
Jul 30 02:34:58.694946 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.694923    2565 generic.go:356] "Generic (PLEG): container finished" podID="db9e89e9-2ea1-42dc-907a-e600187c2eb6" containerID="bb48458e93439a56bec32d19af673f1278a9aa3895729bd0d4e417c9c9470ed4" exitCode=0
Jul 30 02:34:58.695027 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.694967    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"
Jul 30 02:34:58.695074 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.695022    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj" event={"ID":"db9e89e9-2ea1-42dc-907a-e600187c2eb6","Type":"ContainerDied","Data":"bb48458e93439a56bec32d19af673f1278a9aa3895729bd0d4e417c9c9470ed4"}
Jul 30 02:34:58.695074 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.695058    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj" event={"ID":"db9e89e9-2ea1-42dc-907a-e600187c2eb6","Type":"ContainerDied","Data":"fcfe4043e76920564b5be1e1f062a6fcbb6e6569565d2f5cd2b47c8ea1eaa123"}
Jul 30 02:34:58.695141 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.695078    2565 scope.go:117] "RemoveContainer" containerID="bb48458e93439a56bec32d19af673f1278a9aa3895729bd0d4e417c9c9470ed4"
Jul 30 02:34:58.703345 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.703326    2565 scope.go:117] "RemoveContainer" containerID="bb48458e93439a56bec32d19af673f1278a9aa3895729bd0d4e417c9c9470ed4"
Jul 30 02:34:58.703692 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:34:58.703670    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb48458e93439a56bec32d19af673f1278a9aa3895729bd0d4e417c9c9470ed4\": container with ID starting with bb48458e93439a56bec32d19af673f1278a9aa3895729bd0d4e417c9c9470ed4 not found: ID does not exist" containerID="bb48458e93439a56bec32d19af673f1278a9aa3895729bd0d4e417c9c9470ed4"
Jul 30 02:34:58.703786 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.703703    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb48458e93439a56bec32d19af673f1278a9aa3895729bd0d4e417c9c9470ed4"} err="failed to get container status \"bb48458e93439a56bec32d19af673f1278a9aa3895729bd0d4e417c9c9470ed4\": rpc error: code = NotFound desc = could not find container \"bb48458e93439a56bec32d19af673f1278a9aa3895729bd0d4e417c9c9470ed4\": container with ID starting with bb48458e93439a56bec32d19af673f1278a9aa3895729bd0d4e417c9c9470ed4 not found: ID does not exist"
Jul 30 02:34:58.714296 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.714246    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5kntq" podStartSLOduration=2.007664281 podStartE2EDuration="4.714227291s" totalImagesPullingTime="2.70656301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:34:58.712116359 +0000 UTC m=+534.358899768" watchObservedRunningTime="2026-07-30 02:34:58.714227291 +0000 UTC m=+534.361010700"
Jul 30 02:34:58.726426 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.726404    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"]
Jul 30 02:34:58.734698 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.734677    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7jmbj"]
Jul 30 02:34:58.800194 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:58.800150    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db9e89e9-2ea1-42dc-907a-e600187c2eb6" path="/var/lib/kubelet/pods/db9e89e9-2ea1-42dc-907a-e600187c2eb6/volumes"
Jul 30 02:34:59.701992 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:59.701955    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5kntq" event={"ID":"cf4f1044-78f4-4086-aac0-406665379140","Type":"ContainerStarted","Data":"94b3e6099572df5d51488e1aca096d553aba318d9d9ed57503e7aaa3bb852b72"}
Jul 30 02:34:59.719900 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:34:59.719849    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x28m9" podStartSLOduration=2.421656998 podStartE2EDuration="5.719835267s" totalImagesPullingTime="3.298178269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:34:59.71930922 +0000 UTC m=+535.366092625" watchObservedRunningTime="2026-07-30 02:34:59.719835267 +0000 UTC m=+535.366618671"
Jul 30 02:35:00.706675 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:00.706639    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x28m9"
Jul 30 02:35:08.210295 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:08.210254    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:35:08.210559 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:08.210341    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:35:08.210559 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:08.210367    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:35:08.743355 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:08.743302    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl" event={"ID":"0c3ad8b3-6927-4c20-a313-51510a35bf13","Type":"ContainerStarted","Data":"f9fbdbae0dc9595a6240a5a5e1bb88a452424fc2a276acdde78bbc421af6fab1"}
Jul 30 02:35:08.763735 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:08.763683    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl" podStartSLOduration=1.304623614 podStartE2EDuration="15.763668793s" totalImagesPullingTime="14.459045179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:35:08.76267323 +0000 UTC m=+544.409456646" watchObservedRunningTime="2026-07-30 02:35:08.763668793 +0000 UTC m=+544.410452189"
Jul 30 02:35:09.603654 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:09.603622    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:35:09.608221 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:09.608194    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:35:09.747716 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:09.747685    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:35:09.748598 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:09.748583    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl"
Jul 30 02:35:30.711599 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:30.711569    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5kntq"
Jul 30 02:35:31.713448 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:31.713419    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-x28m9"
Jul 30 02:35:44.884764 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:44.884730    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cw5ws"]
Jul 30 02:35:44.885135 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:44.885044    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db9e89e9-2ea1-42dc-907a-e600187c2eb6" containerName="discovery"
Jul 30 02:35:44.885135 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:44.885055    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="db9e89e9-2ea1-42dc-907a-e600187c2eb6" containerName="discovery"
Jul 30 02:35:44.885135 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:44.885113    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="db9e89e9-2ea1-42dc-907a-e600187c2eb6" containerName="discovery"
Jul 30 02:35:44.896863 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:44.896840    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cw5ws"
Jul 30 02:35:44.899836 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:44.899808    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 30 02:35:44.899992 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:44.899808    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 30 02:35:44.899992 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:44.899823    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-t9nl7\""
Jul 30 02:35:44.906186 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:44.906143    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cw5ws"]
Jul 30 02:35:45.061152 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:45.061120    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ct4c\" (UniqueName: \"kubernetes.io/projected/50d3e695-baaf-42fc-aa71-549943a684e2-kube-api-access-5ct4c\") pod \"kuadrant-operator-catalog-cw5ws\" (UID: \"50d3e695-baaf-42fc-aa71-549943a684e2\") " pod="kuadrant-system/kuadrant-operator-catalog-cw5ws"
Jul 30 02:35:45.162256 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:45.162156    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ct4c\" (UniqueName: \"kubernetes.io/projected/50d3e695-baaf-42fc-aa71-549943a684e2-kube-api-access-5ct4c\") pod \"kuadrant-operator-catalog-cw5ws\" (UID: \"50d3e695-baaf-42fc-aa71-549943a684e2\") " pod="kuadrant-system/kuadrant-operator-catalog-cw5ws"
Jul 30 02:35:45.171775 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:45.171746    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ct4c\" (UniqueName: \"kubernetes.io/projected/50d3e695-baaf-42fc-aa71-549943a684e2-kube-api-access-5ct4c\") pod \"kuadrant-operator-catalog-cw5ws\" (UID: \"50d3e695-baaf-42fc-aa71-549943a684e2\") " pod="kuadrant-system/kuadrant-operator-catalog-cw5ws"
Jul 30 02:35:45.207451 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:45.207404    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cw5ws"
Jul 30 02:35:45.242954 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:45.242923    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cw5ws"]
Jul 30 02:35:45.335504 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:45.335476    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cw5ws"]
Jul 30 02:35:45.337332 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:35:45.337307    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d3e695_baaf_42fc_aa71_549943a684e2.slice/crio-97d1f32d8d7185167f0982d85e54f3b50f9c402d228700f04d052283d58dcb47 WatchSource:0}: Error finding container 97d1f32d8d7185167f0982d85e54f3b50f9c402d228700f04d052283d58dcb47: Status 404 returned error can't find the container with id 97d1f32d8d7185167f0982d85e54f3b50f9c402d228700f04d052283d58dcb47
Jul 30 02:35:45.453356 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:45.453276    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4xdbc"]
Jul 30 02:35:45.458157 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:45.458141    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4xdbc"
Jul 30 02:35:45.467246 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:45.467224    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4xdbc"]
Jul 30 02:35:45.566792 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:45.566761    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qtq2\" (UniqueName: \"kubernetes.io/projected/fb3ddd39-1487-46cf-902d-58950c899a5e-kube-api-access-9qtq2\") pod \"kuadrant-operator-catalog-4xdbc\" (UID: \"fb3ddd39-1487-46cf-902d-58950c899a5e\") " pod="kuadrant-system/kuadrant-operator-catalog-4xdbc"
Jul 30 02:35:45.667517 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:45.667485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qtq2\" (UniqueName: \"kubernetes.io/projected/fb3ddd39-1487-46cf-902d-58950c899a5e-kube-api-access-9qtq2\") pod \"kuadrant-operator-catalog-4xdbc\" (UID: \"fb3ddd39-1487-46cf-902d-58950c899a5e\") " pod="kuadrant-system/kuadrant-operator-catalog-4xdbc"
Jul 30 02:35:45.676522 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:45.676488    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qtq2\" (UniqueName: \"kubernetes.io/projected/fb3ddd39-1487-46cf-902d-58950c899a5e-kube-api-access-9qtq2\") pod \"kuadrant-operator-catalog-4xdbc\" (UID: \"fb3ddd39-1487-46cf-902d-58950c899a5e\") " pod="kuadrant-system/kuadrant-operator-catalog-4xdbc"
Jul 30 02:35:45.767659 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:45.767578    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4xdbc"
Jul 30 02:35:45.903120 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:45.903079    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4xdbc"]
Jul 30 02:35:45.907905 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:45.907878    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cw5ws" event={"ID":"50d3e695-baaf-42fc-aa71-549943a684e2","Type":"ContainerStarted","Data":"97d1f32d8d7185167f0982d85e54f3b50f9c402d228700f04d052283d58dcb47"}
Jul 30 02:35:45.908336 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:35:45.908317    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb3ddd39_1487_46cf_902d_58950c899a5e.slice/crio-9f4175bd27b481439253c698d1d4c560b532fba10a9f7400ed168c73c51f50d1 WatchSource:0}: Error finding container 9f4175bd27b481439253c698d1d4c560b532fba10a9f7400ed168c73c51f50d1: Status 404 returned error can't find the container with id 9f4175bd27b481439253c698d1d4c560b532fba10a9f7400ed168c73c51f50d1
Jul 30 02:35:46.914255 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:46.914218    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4xdbc" event={"ID":"fb3ddd39-1487-46cf-902d-58950c899a5e","Type":"ContainerStarted","Data":"9f4175bd27b481439253c698d1d4c560b532fba10a9f7400ed168c73c51f50d1"}
Jul 30 02:35:47.919772 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:47.919733    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4xdbc" event={"ID":"fb3ddd39-1487-46cf-902d-58950c899a5e","Type":"ContainerStarted","Data":"853bc273ec1c625494c7235e4f5477a64cd0b6aeb97d275b867c45c0242bd214"}
Jul 30 02:35:47.921287 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:47.921262    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cw5ws" event={"ID":"50d3e695-baaf-42fc-aa71-549943a684e2","Type":"ContainerStarted","Data":"9e1daaf7046c62a7a312362d5b5c5f325c5e8ffab1f8e670322648bef14ebd46"}
Jul 30 02:35:47.921437 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:47.921344    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-cw5ws" podUID="50d3e695-baaf-42fc-aa71-549943a684e2" containerName="registry-server" containerID="cri-o://9e1daaf7046c62a7a312362d5b5c5f325c5e8ffab1f8e670322648bef14ebd46" gracePeriod=2
Jul 30 02:35:47.937050 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:47.937010    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-4xdbc" podStartSLOduration=1.421214069 podStartE2EDuration="2.936995061s" totalImagesPullingTime="1.515780992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:35:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:35:47.934820227 +0000 UTC m=+583.581603633" watchObservedRunningTime="2026-07-30 02:35:47.936995061 +0000 UTC m=+583.583778466"
Jul 30 02:35:47.951954 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:47.951913    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-cw5ws" podStartSLOduration=1.865600822 podStartE2EDuration="3.951897881s" totalImagesPullingTime="2.086297059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:35:47.951337101 +0000 UTC m=+583.598120505" watchObservedRunningTime="2026-07-30 02:35:47.951897881 +0000 UTC m=+583.598681277"
Jul 30 02:35:48.157573 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:48.157549    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cw5ws"
Jul 30 02:35:48.185495 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:48.185435    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ct4c\" (UniqueName: \"kubernetes.io/projected/50d3e695-baaf-42fc-aa71-549943a684e2-kube-api-access-5ct4c\") pod \"50d3e695-baaf-42fc-aa71-549943a684e2\" (UID: \"50d3e695-baaf-42fc-aa71-549943a684e2\") "
Jul 30 02:35:48.187746 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:48.187716    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d3e695-baaf-42fc-aa71-549943a684e2-kube-api-access-5ct4c" (OuterVolumeSpecName: "kube-api-access-5ct4c") pod "50d3e695-baaf-42fc-aa71-549943a684e2" (UID: "50d3e695-baaf-42fc-aa71-549943a684e2"). InnerVolumeSpecName "kube-api-access-5ct4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:35:48.286003 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:48.285970    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ct4c\" (UniqueName: \"kubernetes.io/projected/50d3e695-baaf-42fc-aa71-549943a684e2-kube-api-access-5ct4c\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:35:48.926948 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:48.926918    2565 generic.go:356] "Generic (PLEG): container finished" podID="50d3e695-baaf-42fc-aa71-549943a684e2" containerID="9e1daaf7046c62a7a312362d5b5c5f325c5e8ffab1f8e670322648bef14ebd46" exitCode=0
Jul 30 02:35:48.927372 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:48.926965    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cw5ws"
Jul 30 02:35:48.927372 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:48.927005    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cw5ws" event={"ID":"50d3e695-baaf-42fc-aa71-549943a684e2","Type":"ContainerDied","Data":"9e1daaf7046c62a7a312362d5b5c5f325c5e8ffab1f8e670322648bef14ebd46"}
Jul 30 02:35:48.927372 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:48.927052    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cw5ws" event={"ID":"50d3e695-baaf-42fc-aa71-549943a684e2","Type":"ContainerDied","Data":"97d1f32d8d7185167f0982d85e54f3b50f9c402d228700f04d052283d58dcb47"}
Jul 30 02:35:48.927372 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:48.927076    2565 scope.go:117] "RemoveContainer" containerID="9e1daaf7046c62a7a312362d5b5c5f325c5e8ffab1f8e670322648bef14ebd46"
Jul 30 02:35:48.935836 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:48.935815    2565 scope.go:117] "RemoveContainer" containerID="9e1daaf7046c62a7a312362d5b5c5f325c5e8ffab1f8e670322648bef14ebd46"
Jul 30 02:35:48.936091 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:35:48.936073    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e1daaf7046c62a7a312362d5b5c5f325c5e8ffab1f8e670322648bef14ebd46\": container with ID starting with 9e1daaf7046c62a7a312362d5b5c5f325c5e8ffab1f8e670322648bef14ebd46 not found: ID does not exist" containerID="9e1daaf7046c62a7a312362d5b5c5f325c5e8ffab1f8e670322648bef14ebd46"
Jul 30 02:35:48.936146 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:48.936100    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e1daaf7046c62a7a312362d5b5c5f325c5e8ffab1f8e670322648bef14ebd46"} err="failed to get container status \"9e1daaf7046c62a7a312362d5b5c5f325c5e8ffab1f8e670322648bef14ebd46\": rpc error: code = NotFound desc = could not find container \"9e1daaf7046c62a7a312362d5b5c5f325c5e8ffab1f8e670322648bef14ebd46\": container with ID starting with 9e1daaf7046c62a7a312362d5b5c5f325c5e8ffab1f8e670322648bef14ebd46 not found: ID does not exist"
Jul 30 02:35:48.942679 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:48.942658    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cw5ws"]
Jul 30 02:35:48.946847 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:48.946827    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cw5ws"]
Jul 30 02:35:50.799975 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:50.799941    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50d3e695-baaf-42fc-aa71-549943a684e2" path="/var/lib/kubelet/pods/50d3e695-baaf-42fc-aa71-549943a684e2/volumes"
Jul 30 02:35:55.768252 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:55.768217    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-4xdbc"
Jul 30 02:35:55.768660 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:55.768367    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-4xdbc"
Jul 30 02:35:55.789832 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:55.789804    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-4xdbc"
Jul 30 02:35:55.979327 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:55.979303    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-4xdbc"
Jul 30 02:35:59.851652 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:59.851616    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v"]
Jul 30 02:35:59.852020 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:59.851938    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d3e695-baaf-42fc-aa71-549943a684e2" containerName="registry-server"
Jul 30 02:35:59.852020 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:59.851949    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d3e695-baaf-42fc-aa71-549943a684e2" containerName="registry-server"
Jul 30 02:35:59.852020 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:59.852006    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d3e695-baaf-42fc-aa71-549943a684e2" containerName="registry-server"
Jul 30 02:35:59.855077 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:59.855061    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v"
Jul 30 02:35:59.857833 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:59.857814    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5gcdd\""
Jul 30 02:35:59.862635 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:59.862613    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v"]
Jul 30 02:35:59.864455 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:59.864432    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1953a83-132c-4c51-ab36-236e674af393-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v\" (UID: \"d1953a83-132c-4c51-ab36-236e674af393\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v"
Jul 30 02:35:59.864574 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:59.864476    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzf2c\" (UniqueName: \"kubernetes.io/projected/d1953a83-132c-4c51-ab36-236e674af393-kube-api-access-zzf2c\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v\" (UID: \"d1953a83-132c-4c51-ab36-236e674af393\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v"
Jul 30 02:35:59.864574 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:59.864507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1953a83-132c-4c51-ab36-236e674af393-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v\" (UID: \"d1953a83-132c-4c51-ab36-236e674af393\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v"
Jul 30 02:35:59.965026 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:59.964993    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1953a83-132c-4c51-ab36-236e674af393-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v\" (UID: \"d1953a83-132c-4c51-ab36-236e674af393\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v"
Jul 30 02:35:59.965231 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:59.965031    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzf2c\" (UniqueName: \"kubernetes.io/projected/d1953a83-132c-4c51-ab36-236e674af393-kube-api-access-zzf2c\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v\" (UID: \"d1953a83-132c-4c51-ab36-236e674af393\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v"
Jul 30 02:35:59.965231 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:59.965060    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1953a83-132c-4c51-ab36-236e674af393-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v\" (UID: \"d1953a83-132c-4c51-ab36-236e674af393\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v"
Jul 30 02:35:59.965404 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:59.965383    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1953a83-132c-4c51-ab36-236e674af393-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v\" (UID: \"d1953a83-132c-4c51-ab36-236e674af393\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v"
Jul 30 02:35:59.965466 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:59.965448    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1953a83-132c-4c51-ab36-236e674af393-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v\" (UID: \"d1953a83-132c-4c51-ab36-236e674af393\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v"
Jul 30 02:35:59.974321 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:35:59.974292    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzf2c\" (UniqueName: \"kubernetes.io/projected/d1953a83-132c-4c51-ab36-236e674af393-kube-api-access-zzf2c\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v\" (UID: \"d1953a83-132c-4c51-ab36-236e674af393\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v"
Jul 30 02:36:00.165105 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.165001    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v"
Jul 30 02:36:00.292223 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.292196    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v"]
Jul 30 02:36:00.294012 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:36:00.293973    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1953a83_132c_4c51_ab36_236e674af393.slice/crio-d927e3ba1fa2f4352329f906c0e8d91e8f9d21f6cdb09392004412c1abdb9ac4 WatchSource:0}: Error finding container d927e3ba1fa2f4352329f906c0e8d91e8f9d21f6cdb09392004412c1abdb9ac4: Status 404 returned error can't find the container with id d927e3ba1fa2f4352329f906c0e8d91e8f9d21f6cdb09392004412c1abdb9ac4
Jul 30 02:36:00.441899 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.441823    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx"]
Jul 30 02:36:00.445094 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.445075    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx"
Jul 30 02:36:00.452784 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.452764    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx"]
Jul 30 02:36:00.467704 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.467677    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c08a7ead-46f4-4139-bc12-4b18542dca53-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx\" (UID: \"c08a7ead-46f4-4139-bc12-4b18542dca53\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx"
Jul 30 02:36:00.467816 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.467749    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv87p\" (UniqueName: \"kubernetes.io/projected/c08a7ead-46f4-4139-bc12-4b18542dca53-kube-api-access-qv87p\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx\" (UID: \"c08a7ead-46f4-4139-bc12-4b18542dca53\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx"
Jul 30 02:36:00.467816 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.467768    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c08a7ead-46f4-4139-bc12-4b18542dca53-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx\" (UID: \"c08a7ead-46f4-4139-bc12-4b18542dca53\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx"
Jul 30 02:36:00.569013 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.568984    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c08a7ead-46f4-4139-bc12-4b18542dca53-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx\" (UID: \"c08a7ead-46f4-4139-bc12-4b18542dca53\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx"
Jul 30 02:36:00.569239 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.569099    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv87p\" (UniqueName: \"kubernetes.io/projected/c08a7ead-46f4-4139-bc12-4b18542dca53-kube-api-access-qv87p\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx\" (UID: \"c08a7ead-46f4-4139-bc12-4b18542dca53\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx"
Jul 30 02:36:00.569239 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.569131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c08a7ead-46f4-4139-bc12-4b18542dca53-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx\" (UID: \"c08a7ead-46f4-4139-bc12-4b18542dca53\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx"
Jul 30 02:36:00.569425 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.569402    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c08a7ead-46f4-4139-bc12-4b18542dca53-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx\" (UID: \"c08a7ead-46f4-4139-bc12-4b18542dca53\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx"
Jul 30 02:36:00.569509 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.569464    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c08a7ead-46f4-4139-bc12-4b18542dca53-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx\" (UID: \"c08a7ead-46f4-4139-bc12-4b18542dca53\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx"
Jul 30 02:36:00.578078 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.578059    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv87p\" (UniqueName: \"kubernetes.io/projected/c08a7ead-46f4-4139-bc12-4b18542dca53-kube-api-access-qv87p\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx\" (UID: \"c08a7ead-46f4-4139-bc12-4b18542dca53\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx"
Jul 30 02:36:00.755767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.755678    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx"
Jul 30 02:36:00.915099 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.915075    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx"]
Jul 30 02:36:00.916464 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:36:00.916440    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08a7ead_46f4_4139_bc12_4b18542dca53.slice/crio-55bac42b0f9f60eaa2f02822e0e5f7fc655a8a55138c40b57f4d5cecbf27ce59 WatchSource:0}: Error finding container 55bac42b0f9f60eaa2f02822e0e5f7fc655a8a55138c40b57f4d5cecbf27ce59: Status 404 returned error can't find the container with id 55bac42b0f9f60eaa2f02822e0e5f7fc655a8a55138c40b57f4d5cecbf27ce59
Jul 30 02:36:00.982201 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.982147    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx" event={"ID":"c08a7ead-46f4-4139-bc12-4b18542dca53","Type":"ContainerStarted","Data":"8226855a2d991d5a61ff74b18722ed4723f28e4ed21fb9db24211457dd8bb905"}
Jul 30 02:36:00.982346 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.982222    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx" event={"ID":"c08a7ead-46f4-4139-bc12-4b18542dca53","Type":"ContainerStarted","Data":"55bac42b0f9f60eaa2f02822e0e5f7fc655a8a55138c40b57f4d5cecbf27ce59"}
Jul 30 02:36:00.983857 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.983831    2565 generic.go:356] "Generic (PLEG): container finished" podID="d1953a83-132c-4c51-ab36-236e674af393" containerID="82d83232157dd7c728181d4a9a9bca2ad6384a34a75a1fd2907b047c27eff950" exitCode=0
Jul 30 02:36:00.983986 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.983910    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v" event={"ID":"d1953a83-132c-4c51-ab36-236e674af393","Type":"ContainerDied","Data":"82d83232157dd7c728181d4a9a9bca2ad6384a34a75a1fd2907b047c27eff950"}
Jul 30 02:36:00.983986 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:00.983938    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v" event={"ID":"d1953a83-132c-4c51-ab36-236e674af393","Type":"ContainerStarted","Data":"d927e3ba1fa2f4352329f906c0e8d91e8f9d21f6cdb09392004412c1abdb9ac4"}
Jul 30 02:36:01.052370 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.052336    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5"]
Jul 30 02:36:01.055724 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.055704    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5"
Jul 30 02:36:01.064543 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.064519    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5"]
Jul 30 02:36:01.072455 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.072435    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d93308e-5b79-4104-a16e-03a7677fed49-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5\" (UID: \"7d93308e-5b79-4104-a16e-03a7677fed49\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5"
Jul 30 02:36:01.072557 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.072484    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d93308e-5b79-4104-a16e-03a7677fed49-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5\" (UID: \"7d93308e-5b79-4104-a16e-03a7677fed49\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5"
Jul 30 02:36:01.072618 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.072607    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ql4h\" (UniqueName: \"kubernetes.io/projected/7d93308e-5b79-4104-a16e-03a7677fed49-kube-api-access-6ql4h\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5\" (UID: \"7d93308e-5b79-4104-a16e-03a7677fed49\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5"
Jul 30 02:36:01.173488 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.173455    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d93308e-5b79-4104-a16e-03a7677fed49-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5\" (UID: \"7d93308e-5b79-4104-a16e-03a7677fed49\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5"
Jul 30 02:36:01.173662 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.173516    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d93308e-5b79-4104-a16e-03a7677fed49-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5\" (UID: \"7d93308e-5b79-4104-a16e-03a7677fed49\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5"
Jul 30 02:36:01.173662 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.173568    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ql4h\" (UniqueName: \"kubernetes.io/projected/7d93308e-5b79-4104-a16e-03a7677fed49-kube-api-access-6ql4h\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5\" (UID: \"7d93308e-5b79-4104-a16e-03a7677fed49\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5"
Jul 30 02:36:01.173986 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.173966    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d93308e-5b79-4104-a16e-03a7677fed49-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5\" (UID: \"7d93308e-5b79-4104-a16e-03a7677fed49\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5"
Jul 30 02:36:01.174065 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.173999    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d93308e-5b79-4104-a16e-03a7677fed49-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5\" (UID: \"7d93308e-5b79-4104-a16e-03a7677fed49\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5"
Jul 30 02:36:01.183357 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.183326    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ql4h\" (UniqueName: \"kubernetes.io/projected/7d93308e-5b79-4104-a16e-03a7677fed49-kube-api-access-6ql4h\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5\" (UID: \"7d93308e-5b79-4104-a16e-03a7677fed49\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5"
Jul 30 02:36:01.372888 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.372854    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5"
Jul 30 02:36:01.455821 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.455791    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r"]
Jul 30 02:36:01.462196 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.462152    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r"
Jul 30 02:36:01.470341 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.470313    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r"]
Jul 30 02:36:01.475703 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.475677    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec4c7f41-225c-43a6-b720-6c566df46dae-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r\" (UID: \"ec4c7f41-225c-43a6-b720-6c566df46dae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r"
Jul 30 02:36:01.475830 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.475754    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwfb2\" (UniqueName: \"kubernetes.io/projected/ec4c7f41-225c-43a6-b720-6c566df46dae-kube-api-access-kwfb2\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r\" (UID: \"ec4c7f41-225c-43a6-b720-6c566df46dae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r"
Jul 30 02:36:01.476398 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.475853    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec4c7f41-225c-43a6-b720-6c566df46dae-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r\" (UID: \"ec4c7f41-225c-43a6-b720-6c566df46dae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r"
Jul 30 02:36:01.508414 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.508389    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5"]
Jul 30 02:36:01.509762 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:36:01.509734    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d93308e_5b79_4104_a16e_03a7677fed49.slice/crio-72e041f88bce3b75bd99bb14bd0bb6775cc1eafaff47c2c88717e7ebc276e65a WatchSource:0}: Error finding container 72e041f88bce3b75bd99bb14bd0bb6775cc1eafaff47c2c88717e7ebc276e65a: Status 404 returned error can't find the container with id 72e041f88bce3b75bd99bb14bd0bb6775cc1eafaff47c2c88717e7ebc276e65a
Jul 30 02:36:01.577105 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.577071    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec4c7f41-225c-43a6-b720-6c566df46dae-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r\" (UID: \"ec4c7f41-225c-43a6-b720-6c566df46dae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r"
Jul 30 02:36:01.577253 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.577136    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwfb2\" (UniqueName: \"kubernetes.io/projected/ec4c7f41-225c-43a6-b720-6c566df46dae-kube-api-access-kwfb2\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r\" (UID: \"ec4c7f41-225c-43a6-b720-6c566df46dae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r"
Jul 30 02:36:01.577253 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.577183    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec4c7f41-225c-43a6-b720-6c566df46dae-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r\" (UID: \"ec4c7f41-225c-43a6-b720-6c566df46dae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r"
Jul 30 02:36:01.577506 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.577484    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec4c7f41-225c-43a6-b720-6c566df46dae-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r\" (UID: \"ec4c7f41-225c-43a6-b720-6c566df46dae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r"
Jul 30 02:36:01.577506 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.577497    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec4c7f41-225c-43a6-b720-6c566df46dae-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r\" (UID: \"ec4c7f41-225c-43a6-b720-6c566df46dae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r"
Jul 30 02:36:01.585875 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.585822    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwfb2\" (UniqueName: \"kubernetes.io/projected/ec4c7f41-225c-43a6-b720-6c566df46dae-kube-api-access-kwfb2\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r\" (UID: \"ec4c7f41-225c-43a6-b720-6c566df46dae\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r"
Jul 30 02:36:01.774866 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.774835    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r"
Jul 30 02:36:01.911692 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.911661    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r"]
Jul 30 02:36:01.913235 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:36:01.913205    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4c7f41_225c_43a6_b720_6c566df46dae.slice/crio-cb5f8245947f54fada2b9f7b14cec3497ab6bdc7df78aa27f0c91ba1a6f31837 WatchSource:0}: Error finding container cb5f8245947f54fada2b9f7b14cec3497ab6bdc7df78aa27f0c91ba1a6f31837: Status 404 returned error can't find the container with id cb5f8245947f54fada2b9f7b14cec3497ab6bdc7df78aa27f0c91ba1a6f31837
Jul 30 02:36:01.990980 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.990935    2565 generic.go:356] "Generic (PLEG): container finished" podID="d1953a83-132c-4c51-ab36-236e674af393" containerID="a21301c9753b773900048cdcde018211ba029a5558a5c2bcdb08980de256be3e" exitCode=0
Jul 30 02:36:01.991513 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.991050    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v" event={"ID":"d1953a83-132c-4c51-ab36-236e674af393","Type":"ContainerDied","Data":"a21301c9753b773900048cdcde018211ba029a5558a5c2bcdb08980de256be3e"}
Jul 30 02:36:01.993022 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.992995    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r" event={"ID":"ec4c7f41-225c-43a6-b720-6c566df46dae","Type":"ContainerStarted","Data":"0acbbc5798d221cfbf1124af97e312937af421f4c1cff92b16ecc75842ae1956"}
Jul 30 02:36:01.993199 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.993048    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r" event={"ID":"ec4c7f41-225c-43a6-b720-6c566df46dae","Type":"ContainerStarted","Data":"cb5f8245947f54fada2b9f7b14cec3497ab6bdc7df78aa27f0c91ba1a6f31837"}
Jul 30 02:36:01.994883 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.994854    2565 generic.go:356] "Generic (PLEG): container finished" podID="7d93308e-5b79-4104-a16e-03a7677fed49" containerID="fde7dea3bee5a6007c20ee16bad26a5b00e38786c0a305b20cb664f96644bb11" exitCode=0
Jul 30 02:36:01.994984 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.994939    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5" event={"ID":"7d93308e-5b79-4104-a16e-03a7677fed49","Type":"ContainerDied","Data":"fde7dea3bee5a6007c20ee16bad26a5b00e38786c0a305b20cb664f96644bb11"}
Jul 30 02:36:01.994984 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.994977    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5" event={"ID":"7d93308e-5b79-4104-a16e-03a7677fed49","Type":"ContainerStarted","Data":"72e041f88bce3b75bd99bb14bd0bb6775cc1eafaff47c2c88717e7ebc276e65a"}
Jul 30 02:36:01.996746 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.996720    2565 generic.go:356] "Generic (PLEG): container finished" podID="c08a7ead-46f4-4139-bc12-4b18542dca53" containerID="8226855a2d991d5a61ff74b18722ed4723f28e4ed21fb9db24211457dd8bb905" exitCode=0
Jul 30 02:36:01.996845 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:01.996766    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx" event={"ID":"c08a7ead-46f4-4139-bc12-4b18542dca53","Type":"ContainerDied","Data":"8226855a2d991d5a61ff74b18722ed4723f28e4ed21fb9db24211457dd8bb905"}
Jul 30 02:36:03.003390 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:03.003346    2565 generic.go:356] "Generic (PLEG): container finished" podID="c08a7ead-46f4-4139-bc12-4b18542dca53" containerID="1cd1efa2911927269ed229224f1f19af398f892b2996570c9fdf91e329f24987" exitCode=0
Jul 30 02:36:03.003811 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:03.003429    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx" event={"ID":"c08a7ead-46f4-4139-bc12-4b18542dca53","Type":"ContainerDied","Data":"1cd1efa2911927269ed229224f1f19af398f892b2996570c9fdf91e329f24987"}
Jul 30 02:36:03.005742 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:03.005706    2565 generic.go:356] "Generic (PLEG): container finished" podID="d1953a83-132c-4c51-ab36-236e674af393" containerID="2afab87403e4fbbdfee77556529db06003433edebade499fd53a494d3d683089" exitCode=0
Jul 30 02:36:03.005834 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:03.005761    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v" event={"ID":"d1953a83-132c-4c51-ab36-236e674af393","Type":"ContainerDied","Data":"2afab87403e4fbbdfee77556529db06003433edebade499fd53a494d3d683089"}
Jul 30 02:36:03.007278 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:03.007257    2565 generic.go:356] "Generic (PLEG): container finished" podID="ec4c7f41-225c-43a6-b720-6c566df46dae" containerID="0acbbc5798d221cfbf1124af97e312937af421f4c1cff92b16ecc75842ae1956" exitCode=0
Jul 30 02:36:03.007378 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:03.007288    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r" event={"ID":"ec4c7f41-225c-43a6-b720-6c566df46dae","Type":"ContainerDied","Data":"0acbbc5798d221cfbf1124af97e312937af421f4c1cff92b16ecc75842ae1956"}
Jul 30 02:36:04.014029 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:04.013946    2565 generic.go:356] "Generic (PLEG): container finished" podID="c08a7ead-46f4-4139-bc12-4b18542dca53" containerID="e88297f61cedb35b98732bad7ae153312274b5bef79bf85971b3c4b0df0edd22" exitCode=0
Jul 30 02:36:04.014432 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:04.014019    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx" event={"ID":"c08a7ead-46f4-4139-bc12-4b18542dca53","Type":"ContainerDied","Data":"e88297f61cedb35b98732bad7ae153312274b5bef79bf85971b3c4b0df0edd22"}
Jul 30 02:36:04.015795 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:04.015772    2565 generic.go:356] "Generic (PLEG): container finished" podID="ec4c7f41-225c-43a6-b720-6c566df46dae" containerID="8d1188efe71937b0177f36a54bfc606055b508d485572a99b9508d48e6eb6ab7" exitCode=0
Jul 30 02:36:04.015911 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:04.015841    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r" event={"ID":"ec4c7f41-225c-43a6-b720-6c566df46dae","Type":"ContainerDied","Data":"8d1188efe71937b0177f36a54bfc606055b508d485572a99b9508d48e6eb6ab7"}
Jul 30 02:36:04.254977 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:04.254953    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v"
Jul 30 02:36:04.300764 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:04.300735    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1953a83-132c-4c51-ab36-236e674af393-bundle\") pod \"d1953a83-132c-4c51-ab36-236e674af393\" (UID: \"d1953a83-132c-4c51-ab36-236e674af393\") "
Jul 30 02:36:04.300916 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:04.300776    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1953a83-132c-4c51-ab36-236e674af393-util\") pod \"d1953a83-132c-4c51-ab36-236e674af393\" (UID: \"d1953a83-132c-4c51-ab36-236e674af393\") "
Jul 30 02:36:04.300916 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:04.300806    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzf2c\" (UniqueName: \"kubernetes.io/projected/d1953a83-132c-4c51-ab36-236e674af393-kube-api-access-zzf2c\") pod \"d1953a83-132c-4c51-ab36-236e674af393\" (UID: \"d1953a83-132c-4c51-ab36-236e674af393\") "
Jul 30 02:36:04.301352 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:04.301321    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1953a83-132c-4c51-ab36-236e674af393-bundle" (OuterVolumeSpecName: "bundle") pod "d1953a83-132c-4c51-ab36-236e674af393" (UID: "d1953a83-132c-4c51-ab36-236e674af393"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:36:04.303021 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:04.303003    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1953a83-132c-4c51-ab36-236e674af393-kube-api-access-zzf2c" (OuterVolumeSpecName: "kube-api-access-zzf2c") pod "d1953a83-132c-4c51-ab36-236e674af393" (UID: "d1953a83-132c-4c51-ab36-236e674af393"). InnerVolumeSpecName "kube-api-access-zzf2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:36:04.306287 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:04.306267    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1953a83-132c-4c51-ab36-236e674af393-util" (OuterVolumeSpecName: "util") pod "d1953a83-132c-4c51-ab36-236e674af393" (UID: "d1953a83-132c-4c51-ab36-236e674af393"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:36:04.401733 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:04.401699    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1953a83-132c-4c51-ab36-236e674af393-bundle\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:36:04.401733 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:04.401726    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1953a83-132c-4c51-ab36-236e674af393-util\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:36:04.401733 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:04.401735    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzf2c\" (UniqueName: \"kubernetes.io/projected/d1953a83-132c-4c51-ab36-236e674af393-kube-api-access-zzf2c\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:36:05.022976 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:05.022883    2565 generic.go:356] "Generic (PLEG): container finished" podID="ec4c7f41-225c-43a6-b720-6c566df46dae" containerID="12ef80668726b9dfbd1ddae5740afaebb2ddbd192bd3152e62133543f0a44cc4" exitCode=0
Jul 30 02:36:05.023452 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:05.022973    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r" event={"ID":"ec4c7f41-225c-43a6-b720-6c566df46dae","Type":"ContainerDied","Data":"12ef80668726b9dfbd1ddae5740afaebb2ddbd192bd3152e62133543f0a44cc4"}
Jul 30 02:36:05.024630 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:05.024610    2565 generic.go:356] "Generic (PLEG): container finished" podID="7d93308e-5b79-4104-a16e-03a7677fed49" containerID="91cb6e89e40bc6917ccf364fd94d77777e9db48e4c1c1d0bec32487b405a48ee" exitCode=0
Jul 30 02:36:05.024732 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:05.024681    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5" event={"ID":"7d93308e-5b79-4104-a16e-03a7677fed49","Type":"ContainerDied","Data":"91cb6e89e40bc6917ccf364fd94d77777e9db48e4c1c1d0bec32487b405a48ee"}
Jul 30 02:36:05.026628 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:05.026608    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v" event={"ID":"d1953a83-132c-4c51-ab36-236e674af393","Type":"ContainerDied","Data":"d927e3ba1fa2f4352329f906c0e8d91e8f9d21f6cdb09392004412c1abdb9ac4"}
Jul 30 02:36:05.026712 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:05.026633    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v"
Jul 30 02:36:05.026712 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:05.026649    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d927e3ba1fa2f4352329f906c0e8d91e8f9d21f6cdb09392004412c1abdb9ac4"
Jul 30 02:36:05.163039 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:05.163011    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx"
Jul 30 02:36:05.210632 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:05.210593    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c08a7ead-46f4-4139-bc12-4b18542dca53-util\") pod \"c08a7ead-46f4-4139-bc12-4b18542dca53\" (UID: \"c08a7ead-46f4-4139-bc12-4b18542dca53\") "
Jul 30 02:36:05.210794 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:05.210655    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c08a7ead-46f4-4139-bc12-4b18542dca53-bundle\") pod \"c08a7ead-46f4-4139-bc12-4b18542dca53\" (UID: \"c08a7ead-46f4-4139-bc12-4b18542dca53\") "
Jul 30 02:36:05.210794 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:05.210730    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv87p\" (UniqueName: \"kubernetes.io/projected/c08a7ead-46f4-4139-bc12-4b18542dca53-kube-api-access-qv87p\") pod \"c08a7ead-46f4-4139-bc12-4b18542dca53\" (UID: \"c08a7ead-46f4-4139-bc12-4b18542dca53\") "
Jul 30 02:36:05.211158 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:05.211134    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08a7ead-46f4-4139-bc12-4b18542dca53-bundle" (OuterVolumeSpecName: "bundle") pod "c08a7ead-46f4-4139-bc12-4b18542dca53" (UID: "c08a7ead-46f4-4139-bc12-4b18542dca53"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:36:05.212915 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:05.212891    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08a7ead-46f4-4139-bc12-4b18542dca53-kube-api-access-qv87p" (OuterVolumeSpecName: "kube-api-access-qv87p") pod "c08a7ead-46f4-4139-bc12-4b18542dca53" (UID: "c08a7ead-46f4-4139-bc12-4b18542dca53"). InnerVolumeSpecName "kube-api-access-qv87p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:36:05.215767 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:05.215747    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08a7ead-46f4-4139-bc12-4b18542dca53-util" (OuterVolumeSpecName: "util") pod "c08a7ead-46f4-4139-bc12-4b18542dca53" (UID: "c08a7ead-46f4-4139-bc12-4b18542dca53"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:36:05.311522 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:05.311484    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c08a7ead-46f4-4139-bc12-4b18542dca53-util\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:36:05.311522 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:05.311514    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c08a7ead-46f4-4139-bc12-4b18542dca53-bundle\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:36:05.311522 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:05.311524    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv87p\" (UniqueName: \"kubernetes.io/projected/c08a7ead-46f4-4139-bc12-4b18542dca53-kube-api-access-qv87p\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:36:06.033500 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:06.033459    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx" event={"ID":"c08a7ead-46f4-4139-bc12-4b18542dca53","Type":"ContainerDied","Data":"55bac42b0f9f60eaa2f02822e0e5f7fc655a8a55138c40b57f4d5cecbf27ce59"}
Jul 30 02:36:06.033927 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:06.033526    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55bac42b0f9f60eaa2f02822e0e5f7fc655a8a55138c40b57f4d5cecbf27ce59"
Jul 30 02:36:06.033927 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:06.033482    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx"
Jul 30 02:36:06.035584 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:06.035556    2565 generic.go:356] "Generic (PLEG): container finished" podID="7d93308e-5b79-4104-a16e-03a7677fed49" containerID="64c74b97b3ac14d9559bb7fb8771bb9459adee6dc11388789d8e2512e853ed8d" exitCode=0
Jul 30 02:36:06.035692 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:06.035590    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5" event={"ID":"7d93308e-5b79-4104-a16e-03a7677fed49","Type":"ContainerDied","Data":"64c74b97b3ac14d9559bb7fb8771bb9459adee6dc11388789d8e2512e853ed8d"}
Jul 30 02:36:06.165139 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:06.165114    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r"
Jul 30 02:36:06.220247 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:06.220211    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwfb2\" (UniqueName: \"kubernetes.io/projected/ec4c7f41-225c-43a6-b720-6c566df46dae-kube-api-access-kwfb2\") pod \"ec4c7f41-225c-43a6-b720-6c566df46dae\" (UID: \"ec4c7f41-225c-43a6-b720-6c566df46dae\") "
Jul 30 02:36:06.220442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:06.220273    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec4c7f41-225c-43a6-b720-6c566df46dae-bundle\") pod \"ec4c7f41-225c-43a6-b720-6c566df46dae\" (UID: \"ec4c7f41-225c-43a6-b720-6c566df46dae\") "
Jul 30 02:36:06.220442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:06.220295    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec4c7f41-225c-43a6-b720-6c566df46dae-util\") pod \"ec4c7f41-225c-43a6-b720-6c566df46dae\" (UID: \"ec4c7f41-225c-43a6-b720-6c566df46dae\") "
Jul 30 02:36:06.220806 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:06.220780    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4c7f41-225c-43a6-b720-6c566df46dae-bundle" (OuterVolumeSpecName: "bundle") pod "ec4c7f41-225c-43a6-b720-6c566df46dae" (UID: "ec4c7f41-225c-43a6-b720-6c566df46dae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:36:06.222576 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:06.222544    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4c7f41-225c-43a6-b720-6c566df46dae-kube-api-access-kwfb2" (OuterVolumeSpecName: "kube-api-access-kwfb2") pod "ec4c7f41-225c-43a6-b720-6c566df46dae" (UID: "ec4c7f41-225c-43a6-b720-6c566df46dae"). InnerVolumeSpecName "kube-api-access-kwfb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:36:06.226156 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:06.226118    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4c7f41-225c-43a6-b720-6c566df46dae-util" (OuterVolumeSpecName: "util") pod "ec4c7f41-225c-43a6-b720-6c566df46dae" (UID: "ec4c7f41-225c-43a6-b720-6c566df46dae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:36:06.321183 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:06.321127    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwfb2\" (UniqueName: \"kubernetes.io/projected/ec4c7f41-225c-43a6-b720-6c566df46dae-kube-api-access-kwfb2\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:36:06.321183 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:06.321197    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec4c7f41-225c-43a6-b720-6c566df46dae-bundle\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:36:06.321394 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:06.321209    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec4c7f41-225c-43a6-b720-6c566df46dae-util\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:36:07.042957 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:07.042931    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r"
Jul 30 02:36:07.043374 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:07.042965    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r" event={"ID":"ec4c7f41-225c-43a6-b720-6c566df46dae","Type":"ContainerDied","Data":"cb5f8245947f54fada2b9f7b14cec3497ab6bdc7df78aa27f0c91ba1a6f31837"}
Jul 30 02:36:07.043374 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:07.043008    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb5f8245947f54fada2b9f7b14cec3497ab6bdc7df78aa27f0c91ba1a6f31837"
Jul 30 02:36:07.176884 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:07.176861    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5"
Jul 30 02:36:07.230558 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:07.230523    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d93308e-5b79-4104-a16e-03a7677fed49-util\") pod \"7d93308e-5b79-4104-a16e-03a7677fed49\" (UID: \"7d93308e-5b79-4104-a16e-03a7677fed49\") "
Jul 30 02:36:07.230751 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:07.230596    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ql4h\" (UniqueName: \"kubernetes.io/projected/7d93308e-5b79-4104-a16e-03a7677fed49-kube-api-access-6ql4h\") pod \"7d93308e-5b79-4104-a16e-03a7677fed49\" (UID: \"7d93308e-5b79-4104-a16e-03a7677fed49\") "
Jul 30 02:36:07.230751 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:07.230639    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d93308e-5b79-4104-a16e-03a7677fed49-bundle\") pod \"7d93308e-5b79-4104-a16e-03a7677fed49\" (UID: \"7d93308e-5b79-4104-a16e-03a7677fed49\") "
Jul 30 02:36:07.231265 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:07.231236    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d93308e-5b79-4104-a16e-03a7677fed49-bundle" (OuterVolumeSpecName: "bundle") pod "7d93308e-5b79-4104-a16e-03a7677fed49" (UID: "7d93308e-5b79-4104-a16e-03a7677fed49"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:36:07.232789 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:07.232764    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d93308e-5b79-4104-a16e-03a7677fed49-kube-api-access-6ql4h" (OuterVolumeSpecName: "kube-api-access-6ql4h") pod "7d93308e-5b79-4104-a16e-03a7677fed49" (UID: "7d93308e-5b79-4104-a16e-03a7677fed49"). InnerVolumeSpecName "kube-api-access-6ql4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:36:07.234910 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:07.234887    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d93308e-5b79-4104-a16e-03a7677fed49-util" (OuterVolumeSpecName: "util") pod "7d93308e-5b79-4104-a16e-03a7677fed49" (UID: "7d93308e-5b79-4104-a16e-03a7677fed49"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:36:07.332005 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:07.331960    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d93308e-5b79-4104-a16e-03a7677fed49-util\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:36:07.332005 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:07.331998    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ql4h\" (UniqueName: \"kubernetes.io/projected/7d93308e-5b79-4104-a16e-03a7677fed49-kube-api-access-6ql4h\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:36:07.332005 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:07.332010    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d93308e-5b79-4104-a16e-03a7677fed49-bundle\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:36:08.049593 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:08.049560    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5"
Jul 30 02:36:08.049593 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:08.049575    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5" event={"ID":"7d93308e-5b79-4104-a16e-03a7677fed49","Type":"ContainerDied","Data":"72e041f88bce3b75bd99bb14bd0bb6775cc1eafaff47c2c88717e7ebc276e65a"}
Jul 30 02:36:08.050046 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:08.049620    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72e041f88bce3b75bd99bb14bd0bb6775cc1eafaff47c2c88717e7ebc276e65a"
Jul 30 02:36:22.483777 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.483739    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9xwx"]
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484036    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c08a7ead-46f4-4139-bc12-4b18542dca53" containerName="util"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484046    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c08a7ead-46f4-4139-bc12-4b18542dca53" containerName="util"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484055    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c08a7ead-46f4-4139-bc12-4b18542dca53" containerName="pull"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484060    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c08a7ead-46f4-4139-bc12-4b18542dca53" containerName="pull"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484066    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c08a7ead-46f4-4139-bc12-4b18542dca53" containerName="extract"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484072    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c08a7ead-46f4-4139-bc12-4b18542dca53" containerName="extract"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484079    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1953a83-132c-4c51-ab36-236e674af393" containerName="extract"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484084    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1953a83-132c-4c51-ab36-236e674af393" containerName="extract"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484090    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d93308e-5b79-4104-a16e-03a7677fed49" containerName="util"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484095    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d93308e-5b79-4104-a16e-03a7677fed49" containerName="util"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484102    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec4c7f41-225c-43a6-b720-6c566df46dae" containerName="util"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484107    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec4c7f41-225c-43a6-b720-6c566df46dae" containerName="util"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484116    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d93308e-5b79-4104-a16e-03a7677fed49" containerName="pull"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484120    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d93308e-5b79-4104-a16e-03a7677fed49" containerName="pull"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484126    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1953a83-132c-4c51-ab36-236e674af393" containerName="pull"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484131    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1953a83-132c-4c51-ab36-236e674af393" containerName="pull"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484139    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec4c7f41-225c-43a6-b720-6c566df46dae" containerName="extract"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484146    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec4c7f41-225c-43a6-b720-6c566df46dae" containerName="extract"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484152    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec4c7f41-225c-43a6-b720-6c566df46dae" containerName="pull"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484156    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec4c7f41-225c-43a6-b720-6c566df46dae" containerName="pull"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484163    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1953a83-132c-4c51-ab36-236e674af393" containerName="util"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484182    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1953a83-132c-4c51-ab36-236e674af393" containerName="util"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484187    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d93308e-5b79-4104-a16e-03a7677fed49" containerName="extract"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484192    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d93308e-5b79-4104-a16e-03a7677fed49" containerName="extract"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484241    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec4c7f41-225c-43a6-b720-6c566df46dae" containerName="extract"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484251    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1953a83-132c-4c51-ab36-236e674af393" containerName="extract"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484257    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c08a7ead-46f4-4139-bc12-4b18542dca53" containerName="extract"
Jul 30 02:36:22.490964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.484263    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d93308e-5b79-4104-a16e-03a7677fed49" containerName="extract"
Jul 30 02:36:22.491815 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.491798    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9xwx"
Jul 30 02:36:22.495089 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.495067    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-jb7hr\""
Jul 30 02:36:22.495246 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.495143    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 30 02:36:22.502517 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.502493    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9xwx"]
Jul 30 02:36:22.556590 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.556560    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7p5z\" (UniqueName: \"kubernetes.io/projected/703264f4-d1c9-4228-8600-11a897120f05-kube-api-access-s7p5z\") pod \"dns-operator-controller-manager-864dd78c4c-n9xwx\" (UID: \"703264f4-d1c9-4228-8600-11a897120f05\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9xwx"
Jul 30 02:36:22.657231 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.657190    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7p5z\" (UniqueName: \"kubernetes.io/projected/703264f4-d1c9-4228-8600-11a897120f05-kube-api-access-s7p5z\") pod \"dns-operator-controller-manager-864dd78c4c-n9xwx\" (UID: \"703264f4-d1c9-4228-8600-11a897120f05\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9xwx"
Jul 30 02:36:22.666083 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.666054    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7p5z\" (UniqueName: \"kubernetes.io/projected/703264f4-d1c9-4228-8600-11a897120f05-kube-api-access-s7p5z\") pod \"dns-operator-controller-manager-864dd78c4c-n9xwx\" (UID: \"703264f4-d1c9-4228-8600-11a897120f05\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9xwx"
Jul 30 02:36:22.801492 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.801466    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9xwx"
Jul 30 02:36:22.936859 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:22.936835    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9xwx"]
Jul 30 02:36:23.118700 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:23.118616    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9xwx" event={"ID":"703264f4-d1c9-4228-8600-11a897120f05","Type":"ContainerStarted","Data":"5985a20e21615a32c9c4cc23089457b846aeeb4e04b0297831b830cae255b302"}
Jul 30 02:36:25.130156 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:25.130125    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9xwx" event={"ID":"703264f4-d1c9-4228-8600-11a897120f05","Type":"ContainerStarted","Data":"de32cc55dbe4796e8cb4a5f1933fe88fc38aad2d7db3a85a47c9910b33cad8de"}
Jul 30 02:36:25.130552 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:25.130228    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9xwx"
Jul 30 02:36:25.151425 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:25.151375    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9xwx" podStartSLOduration=1.433006229 podStartE2EDuration="3.151359588s" totalImagesPullingTime="1.718353359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:36:25.150126432 +0000 UTC m=+620.796909837" watchObservedRunningTime="2026-07-30 02:36:25.151359588 +0000 UTC m=+620.798143033"
Jul 30 02:36:25.922286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:25.922195    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"]
Jul 30 02:36:25.925663 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:25.925642    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"
Jul 30 02:36:25.928624 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:25.928605    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-pkxld\""
Jul 30 02:36:25.935110 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:25.935086    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"]
Jul 30 02:36:25.988356 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:25.988330    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wx52\" (UniqueName: \"kubernetes.io/projected/fdbd7ae8-ae26-4213-bdf1-f75542880464-kube-api-access-4wx52\") pod \"limitador-operator-controller-manager-859d7b78bd-z8ck5\" (UID: \"fdbd7ae8-ae26-4213-bdf1-f75542880464\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"
Jul 30 02:36:26.088719 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:26.088691    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wx52\" (UniqueName: \"kubernetes.io/projected/fdbd7ae8-ae26-4213-bdf1-f75542880464-kube-api-access-4wx52\") pod \"limitador-operator-controller-manager-859d7b78bd-z8ck5\" (UID: \"fdbd7ae8-ae26-4213-bdf1-f75542880464\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"
Jul 30 02:36:26.097678 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:26.097647    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wx52\" (UniqueName: \"kubernetes.io/projected/fdbd7ae8-ae26-4213-bdf1-f75542880464-kube-api-access-4wx52\") pod \"limitador-operator-controller-manager-859d7b78bd-z8ck5\" (UID: \"fdbd7ae8-ae26-4213-bdf1-f75542880464\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"
Jul 30 02:36:26.235944 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:26.235860    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"
Jul 30 02:36:26.369570 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:26.369542    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"]
Jul 30 02:36:26.371507 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:36:26.371482    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbd7ae8_ae26_4213_bdf1_f75542880464.slice/crio-bc159ccf4a105a4f1f6609353de007604cd522cbf71ba3960b40b237569e6eee WatchSource:0}: Error finding container bc159ccf4a105a4f1f6609353de007604cd522cbf71ba3960b40b237569e6eee: Status 404 returned error can't find the container with id bc159ccf4a105a4f1f6609353de007604cd522cbf71ba3960b40b237569e6eee
Jul 30 02:36:27.141529 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:27.141495    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5" event={"ID":"fdbd7ae8-ae26-4213-bdf1-f75542880464","Type":"ContainerStarted","Data":"bc159ccf4a105a4f1f6609353de007604cd522cbf71ba3960b40b237569e6eee"}
Jul 30 02:36:29.153808 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:29.153768    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5" event={"ID":"fdbd7ae8-ae26-4213-bdf1-f75542880464","Type":"ContainerStarted","Data":"6d9bec19578198eccea0bd22ca164f6470d40d06719b4dd882b9da0058e49522"}
Jul 30 02:36:29.154233 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:29.153868    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"
Jul 30 02:36:29.172582 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:29.172529    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5" podStartSLOduration=2.281574389 podStartE2EDuration="4.172515594s" totalImagesPullingTime="1.890941205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:36:29.171582721 +0000 UTC m=+624.818366128" watchObservedRunningTime="2026-07-30 02:36:29.172515594 +0000 UTC m=+624.819298999"
Jul 30 02:36:29.792676 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:29.792639    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-b265s"]
Jul 30 02:36:29.796159 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:29.796132    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b265s"
Jul 30 02:36:29.800205 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:29.800183    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-kd54f\""
Jul 30 02:36:29.808316 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:29.808289    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-b265s"]
Jul 30 02:36:29.929882 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:29.929840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbsjt\" (UniqueName: \"kubernetes.io/projected/8b2ce1d9-ce1b-49e6-91f8-92c11bcba18c-kube-api-access-cbsjt\") pod \"authorino-operator-75cc8d5cb7-b265s\" (UID: \"8b2ce1d9-ce1b-49e6-91f8-92c11bcba18c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-b265s"
Jul 30 02:36:30.030436 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:30.030404    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbsjt\" (UniqueName: \"kubernetes.io/projected/8b2ce1d9-ce1b-49e6-91f8-92c11bcba18c-kube-api-access-cbsjt\") pod \"authorino-operator-75cc8d5cb7-b265s\" (UID: \"8b2ce1d9-ce1b-49e6-91f8-92c11bcba18c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-b265s"
Jul 30 02:36:30.043905 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:30.043833    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbsjt\" (UniqueName: \"kubernetes.io/projected/8b2ce1d9-ce1b-49e6-91f8-92c11bcba18c-kube-api-access-cbsjt\") pod \"authorino-operator-75cc8d5cb7-b265s\" (UID: \"8b2ce1d9-ce1b-49e6-91f8-92c11bcba18c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-b265s"
Jul 30 02:36:30.106745 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:30.106712    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b265s"
Jul 30 02:36:30.240139 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:30.240108    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-b265s"]
Jul 30 02:36:30.242245 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:36:30.242219    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2ce1d9_ce1b_49e6_91f8_92c11bcba18c.slice/crio-154955fc38c0cdefc0d0fd35cfbf1ba1338103d2209d2f879f543fe5e780a876 WatchSource:0}: Error finding container 154955fc38c0cdefc0d0fd35cfbf1ba1338103d2209d2f879f543fe5e780a876: Status 404 returned error can't find the container with id 154955fc38c0cdefc0d0fd35cfbf1ba1338103d2209d2f879f543fe5e780a876
Jul 30 02:36:31.168820 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:31.168782    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b265s" event={"ID":"8b2ce1d9-ce1b-49e6-91f8-92c11bcba18c","Type":"ContainerStarted","Data":"154955fc38c0cdefc0d0fd35cfbf1ba1338103d2209d2f879f543fe5e780a876"}
Jul 30 02:36:33.181234 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:33.181192    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b265s" event={"ID":"8b2ce1d9-ce1b-49e6-91f8-92c11bcba18c","Type":"ContainerStarted","Data":"1720cef1ae4b181b048dba0326c7870afe0db7f43b6426759e23370eb979468e"}
Jul 30 02:36:33.181694 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:33.181300    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b265s"
Jul 30 02:36:33.203152 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:33.203101    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b265s" podStartSLOduration=1.9045364550000001 podStartE2EDuration="4.203086232s" totalImagesPullingTime="2.298549777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:36:33.201219082 +0000 UTC m=+628.848002487" watchObservedRunningTime="2026-07-30 02:36:33.203086232 +0000 UTC m=+628.849869637"
Jul 30 02:36:34.844066 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:34.844029    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"]
Jul 30 02:36:34.847520 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:34.847496    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"
Jul 30 02:36:34.850771 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:34.850749    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kgss5\""
Jul 30 02:36:34.858833 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:34.858812    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"]
Jul 30 02:36:34.978262 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:34.978227    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knj4t\" (UniqueName: \"kubernetes.io/projected/12569a05-86c2-4146-86c4-6e09bb663d60-kube-api-access-knj4t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wqt8k\" (UID: \"12569a05-86c2-4146-86c4-6e09bb663d60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"
Jul 30 02:36:34.978469 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:34.978293    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12569a05-86c2-4146-86c4-6e09bb663d60-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wqt8k\" (UID: \"12569a05-86c2-4146-86c4-6e09bb663d60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"
Jul 30 02:36:35.079662 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:35.079625    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12569a05-86c2-4146-86c4-6e09bb663d60-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wqt8k\" (UID: \"12569a05-86c2-4146-86c4-6e09bb663d60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"
Jul 30 02:36:35.079852 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:35.079715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knj4t\" (UniqueName: \"kubernetes.io/projected/12569a05-86c2-4146-86c4-6e09bb663d60-kube-api-access-knj4t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wqt8k\" (UID: \"12569a05-86c2-4146-86c4-6e09bb663d60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"
Jul 30 02:36:35.080021 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:35.079999    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12569a05-86c2-4146-86c4-6e09bb663d60-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wqt8k\" (UID: \"12569a05-86c2-4146-86c4-6e09bb663d60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"
Jul 30 02:36:35.088500 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:35.088475    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knj4t\" (UniqueName: \"kubernetes.io/projected/12569a05-86c2-4146-86c4-6e09bb663d60-kube-api-access-knj4t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wqt8k\" (UID: \"12569a05-86c2-4146-86c4-6e09bb663d60\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"
Jul 30 02:36:35.159396 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:35.159307    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"
Jul 30 02:36:35.294359 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:35.294333    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"]
Jul 30 02:36:35.295836 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:36:35.295814    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12569a05_86c2_4146_86c4_6e09bb663d60.slice/crio-296e99f2b99955d5507c9976140dcc5b2d1bb85fa2aaecb4edcbed55bea86a48 WatchSource:0}: Error finding container 296e99f2b99955d5507c9976140dcc5b2d1bb85fa2aaecb4edcbed55bea86a48: Status 404 returned error can't find the container with id 296e99f2b99955d5507c9976140dcc5b2d1bb85fa2aaecb4edcbed55bea86a48
Jul 30 02:36:36.137633 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:36.137604    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n9xwx"
Jul 30 02:36:36.199975 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:36.199939    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k" event={"ID":"12569a05-86c2-4146-86c4-6e09bb663d60","Type":"ContainerStarted","Data":"296e99f2b99955d5507c9976140dcc5b2d1bb85fa2aaecb4edcbed55bea86a48"}
Jul 30 02:36:40.163154 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:40.163124    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"
Jul 30 02:36:40.227756 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:40.227707    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k" event={"ID":"12569a05-86c2-4146-86c4-6e09bb663d60","Type":"ContainerStarted","Data":"584b939da10e238bb5679b9d7b78edf44ff5134caebb283e9d8652c8087c4464"}
Jul 30 02:36:40.227918 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:40.227868    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"
Jul 30 02:36:40.250898 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:40.250845    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k" podStartSLOduration=1.5283884410000002 podStartE2EDuration="6.25082953s" totalImagesPullingTime="4.722441089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:36:40.249496019 +0000 UTC m=+635.896279426" watchObservedRunningTime="2026-07-30 02:36:40.25082953 +0000 UTC m=+635.897612935"
Jul 30 02:36:42.678658 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:42.678624    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zsg96"]
Jul 30 02:36:42.682315 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:42.682292    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zsg96"
Jul 30 02:36:42.687253 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:42.687233    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 30 02:36:42.687392 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:42.687279    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 30 02:36:42.687392 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:42.687365    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5gcdd\""
Jul 30 02:36:42.691710 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:42.691689    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zsg96"]
Jul 30 02:36:42.741892 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:42.741860    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xkp9\" (UniqueName: \"kubernetes.io/projected/f8fe9862-0773-4e89-b2b3-d4bd37967bbb-kube-api-access-7xkp9\") pod \"kuadrant-console-plugin-66b5774985-zsg96\" (UID: \"f8fe9862-0773-4e89-b2b3-d4bd37967bbb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zsg96"
Jul 30 02:36:42.742116 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:42.741928    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f8fe9862-0773-4e89-b2b3-d4bd37967bbb-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zsg96\" (UID: \"f8fe9862-0773-4e89-b2b3-d4bd37967bbb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zsg96"
Jul 30 02:36:42.742116 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:42.741990    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8fe9862-0773-4e89-b2b3-d4bd37967bbb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zsg96\" (UID: \"f8fe9862-0773-4e89-b2b3-d4bd37967bbb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zsg96"
Jul 30 02:36:42.842568 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:42.842536    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8fe9862-0773-4e89-b2b3-d4bd37967bbb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zsg96\" (UID: \"f8fe9862-0773-4e89-b2b3-d4bd37967bbb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zsg96"
Jul 30 02:36:42.842757 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:42.842576    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xkp9\" (UniqueName: \"kubernetes.io/projected/f8fe9862-0773-4e89-b2b3-d4bd37967bbb-kube-api-access-7xkp9\") pod \"kuadrant-console-plugin-66b5774985-zsg96\" (UID: \"f8fe9862-0773-4e89-b2b3-d4bd37967bbb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zsg96"
Jul 30 02:36:42.842757 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:42.842608    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f8fe9862-0773-4e89-b2b3-d4bd37967bbb-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zsg96\" (UID: \"f8fe9862-0773-4e89-b2b3-d4bd37967bbb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zsg96"
Jul 30 02:36:42.842757 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:36:42.842703    2565 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 30 02:36:42.842929 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:36:42.842779    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8fe9862-0773-4e89-b2b3-d4bd37967bbb-plugin-serving-cert podName:f8fe9862-0773-4e89-b2b3-d4bd37967bbb nodeName:}" failed. No retries permitted until 2026-07-30 02:36:43.342757229 +0000 UTC m=+638.989540612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/f8fe9862-0773-4e89-b2b3-d4bd37967bbb-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-zsg96" (UID: "f8fe9862-0773-4e89-b2b3-d4bd37967bbb") : secret "plugin-serving-cert" not found
Jul 30 02:36:42.843228 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:42.843212    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f8fe9862-0773-4e89-b2b3-d4bd37967bbb-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zsg96\" (UID: \"f8fe9862-0773-4e89-b2b3-d4bd37967bbb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zsg96"
Jul 30 02:36:42.852609 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:42.852589    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xkp9\" (UniqueName: \"kubernetes.io/projected/f8fe9862-0773-4e89-b2b3-d4bd37967bbb-kube-api-access-7xkp9\") pod \"kuadrant-console-plugin-66b5774985-zsg96\" (UID: \"f8fe9862-0773-4e89-b2b3-d4bd37967bbb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zsg96"
Jul 30 02:36:43.345979 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:43.345941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8fe9862-0773-4e89-b2b3-d4bd37967bbb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zsg96\" (UID: \"f8fe9862-0773-4e89-b2b3-d4bd37967bbb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zsg96"
Jul 30 02:36:43.348487 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:43.348462    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8fe9862-0773-4e89-b2b3-d4bd37967bbb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zsg96\" (UID: \"f8fe9862-0773-4e89-b2b3-d4bd37967bbb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zsg96"
Jul 30 02:36:43.591811 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:43.591776    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zsg96"
Jul 30 02:36:43.750401 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:43.750370    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zsg96"]
Jul 30 02:36:43.750722 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:36:43.750700    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8fe9862_0773_4e89_b2b3_d4bd37967bbb.slice/crio-897af0833afdc7daec289fa28b2660e6b06ee048754f4b99f21eb8488765a6ad WatchSource:0}: Error finding container 897af0833afdc7daec289fa28b2660e6b06ee048754f4b99f21eb8488765a6ad: Status 404 returned error can't find the container with id 897af0833afdc7daec289fa28b2660e6b06ee048754f4b99f21eb8488765a6ad
Jul 30 02:36:44.188904 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:44.188870    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b265s"
Jul 30 02:36:44.248393 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:44.248360    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zsg96" event={"ID":"f8fe9862-0773-4e89-b2b3-d4bd37967bbb","Type":"ContainerStarted","Data":"897af0833afdc7daec289fa28b2660e6b06ee048754f4b99f21eb8488765a6ad"}
Jul 30 02:36:51.235672 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:51.235640    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"
Jul 30 02:36:52.959089 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:52.959055    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"]
Jul 30 02:36:52.959568 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:52.959300    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k" podUID="12569a05-86c2-4146-86c4-6e09bb663d60" containerName="manager" containerID="cri-o://584b939da10e238bb5679b9d7b78edf44ff5134caebb283e9d8652c8087c4464" gracePeriod=2
Jul 30 02:36:52.969138 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:52.969106    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"]
Jul 30 02:36:52.973186 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:36:52.973133    2565 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"12569a05-86c2-4146-86c4-6e09bb663d60\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-wqt8k\": pods \"kuadrant-operator-controller-manager-8684c9ff49-wqt8k\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"
Jul 30 02:36:52.980197 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:52.980156    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"]
Jul 30 02:36:52.980444 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:52.980415    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5" podUID="fdbd7ae8-ae26-4213-bdf1-f75542880464" containerName="manager" containerID="cri-o://6d9bec19578198eccea0bd22ca164f6470d40d06719b4dd882b9da0058e49522" gracePeriod=2
Jul 30 02:36:52.983436 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:36:52.983401    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wqt8k\" is forbidden: User \"system:node:ip-10-0-132-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-89.ec2.internal' and this object" podUID="12569a05-86c2-4146-86c4-6e09bb663d60" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"
Jul 30 02:36:52.995206 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:52.994897    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll"]
Jul 30 02:36:52.996496 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:52.995575    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12569a05-86c2-4146-86c4-6e09bb663d60" containerName="manager"
Jul 30 02:36:52.996496 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:52.995596    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="12569a05-86c2-4146-86c4-6e09bb663d60" containerName="manager"
Jul 30 02:36:52.996496 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:52.995680    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="12569a05-86c2-4146-86c4-6e09bb663d60" containerName="manager"
Jul 30 02:36:52.997857 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:36:52.997817    2565 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"fdbd7ae8-ae26-4213-bdf1-f75542880464\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-z8ck5\": pods \"limitador-operator-controller-manager-859d7b78bd-z8ck5\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"
Jul 30 02:36:52.999009 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:52.998989    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"]
Jul 30 02:36:52.999153 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:52.999072    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll"
Jul 30 02:36:53.003088 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:36:53.003056    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wqt8k\" is forbidden: User \"system:node:ip-10-0-132-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-89.ec2.internal' and this object" podUID="12569a05-86c2-4146-86c4-6e09bb663d60" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"
Jul 30 02:36:53.009897 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:53.009875    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll"]
Jul 30 02:36:53.035340 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:53.035305    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zfbll\" (UID: \"79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll"
Jul 30 02:36:53.035480 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:53.035452    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnztr\" (UniqueName: \"kubernetes.io/projected/79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8-kube-api-access-jnztr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zfbll\" (UID: \"79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll"
Jul 30 02:36:53.044220 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:36:53.044162    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z8ck5\" is forbidden: User \"system:node:ip-10-0-132-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-89.ec2.internal' and this object" podUID="fdbd7ae8-ae26-4213-bdf1-f75542880464" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"
Jul 30 02:36:53.136119 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:53.136082    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnztr\" (UniqueName: \"kubernetes.io/projected/79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8-kube-api-access-jnztr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zfbll\" (UID: \"79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll"
Jul 30 02:36:53.136321 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:53.136195    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zfbll\" (UID: \"79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll"
Jul 30 02:36:53.136571 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:53.136545    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zfbll\" (UID: \"79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll"
Jul 30 02:36:53.145348 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:53.145322    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnztr\" (UniqueName: \"kubernetes.io/projected/79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8-kube-api-access-jnztr\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zfbll\" (UID: \"79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll"
Jul 30 02:36:53.386068 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:36:53.386026    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll"
Jul 30 02:36:54.801814 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:36:54.801764    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z8ck5\" is forbidden: User \"system:node:ip-10-0-132-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-89.ec2.internal' and this object" podUID="fdbd7ae8-ae26-4213-bdf1-f75542880464" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"
Jul 30 02:36:54.804839 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:36:54.804791    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wqt8k\" is forbidden: User \"system:node:ip-10-0-132-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-89.ec2.internal' and this object" podUID="12569a05-86c2-4146-86c4-6e09bb663d60" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"
Jul 30 02:37:05.004745 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.004719    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"
Jul 30 02:37:05.007153 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:37:05.007125    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z8ck5\" is forbidden: User \"system:node:ip-10-0-132-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-89.ec2.internal' and this object" podUID="fdbd7ae8-ae26-4213-bdf1-f75542880464" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"
Jul 30 02:37:05.019015 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.018992    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll"]
Jul 30 02:37:05.019404 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:37:05.019384    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79cf9fd9_3894_4fc3_a5cc_a8b04511e1e8.slice/crio-55b02b5fb26b22d2d2cda3a6c8f8bc6205f304243442517373f666028a87eeaa WatchSource:0}: Error finding container 55b02b5fb26b22d2d2cda3a6c8f8bc6205f304243442517373f666028a87eeaa: Status 404 returned error can't find the container with id 55b02b5fb26b22d2d2cda3a6c8f8bc6205f304243442517373f666028a87eeaa
Jul 30 02:37:05.020330 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.020309    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"
Jul 30 02:37:05.022918 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:37:05.022891    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wqt8k\" is forbidden: User \"system:node:ip-10-0-132-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-89.ec2.internal' and this object" podUID="12569a05-86c2-4146-86c4-6e09bb663d60" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"
Jul 30 02:37:05.025299 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:37:05.025276    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z8ck5\" is forbidden: User \"system:node:ip-10-0-132-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-89.ec2.internal' and this object" podUID="fdbd7ae8-ae26-4213-bdf1-f75542880464" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"
Jul 30 02:37:05.146711 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.146634    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12569a05-86c2-4146-86c4-6e09bb663d60-extensions-socket-volume\") pod \"12569a05-86c2-4146-86c4-6e09bb663d60\" (UID: \"12569a05-86c2-4146-86c4-6e09bb663d60\") "
Jul 30 02:37:05.146711 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.146669    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knj4t\" (UniqueName: \"kubernetes.io/projected/12569a05-86c2-4146-86c4-6e09bb663d60-kube-api-access-knj4t\") pod \"12569a05-86c2-4146-86c4-6e09bb663d60\" (UID: \"12569a05-86c2-4146-86c4-6e09bb663d60\") "
Jul 30 02:37:05.146711 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.146686    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wx52\" (UniqueName: \"kubernetes.io/projected/fdbd7ae8-ae26-4213-bdf1-f75542880464-kube-api-access-4wx52\") pod \"fdbd7ae8-ae26-4213-bdf1-f75542880464\" (UID: \"fdbd7ae8-ae26-4213-bdf1-f75542880464\") "
Jul 30 02:37:05.147216 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.147161    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12569a05-86c2-4146-86c4-6e09bb663d60-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "12569a05-86c2-4146-86c4-6e09bb663d60" (UID: "12569a05-86c2-4146-86c4-6e09bb663d60"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:37:05.148794 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.148770    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbd7ae8-ae26-4213-bdf1-f75542880464-kube-api-access-4wx52" (OuterVolumeSpecName: "kube-api-access-4wx52") pod "fdbd7ae8-ae26-4213-bdf1-f75542880464" (UID: "fdbd7ae8-ae26-4213-bdf1-f75542880464"). InnerVolumeSpecName "kube-api-access-4wx52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:37:05.148885 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.148835    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12569a05-86c2-4146-86c4-6e09bb663d60-kube-api-access-knj4t" (OuterVolumeSpecName: "kube-api-access-knj4t") pod "12569a05-86c2-4146-86c4-6e09bb663d60" (UID: "12569a05-86c2-4146-86c4-6e09bb663d60"). InnerVolumeSpecName "kube-api-access-knj4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:37:05.247546 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.247514    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12569a05-86c2-4146-86c4-6e09bb663d60-extensions-socket-volume\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:37:05.247546 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.247544    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knj4t\" (UniqueName: \"kubernetes.io/projected/12569a05-86c2-4146-86c4-6e09bb663d60-kube-api-access-knj4t\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:37:05.247742 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.247559    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wx52\" (UniqueName: \"kubernetes.io/projected/fdbd7ae8-ae26-4213-bdf1-f75542880464-kube-api-access-4wx52\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:37:05.373924 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.373870    2565 generic.go:356] "Generic (PLEG): container finished" podID="fdbd7ae8-ae26-4213-bdf1-f75542880464" containerID="6d9bec19578198eccea0bd22ca164f6470d40d06719b4dd882b9da0058e49522" exitCode=0
Jul 30 02:37:05.374112 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.373941    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"
Jul 30 02:37:05.374112 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.373957    2565 scope.go:117] "RemoveContainer" containerID="6d9bec19578198eccea0bd22ca164f6470d40d06719b4dd882b9da0058e49522"
Jul 30 02:37:05.376227 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.376201    2565 generic.go:356] "Generic (PLEG): container finished" podID="12569a05-86c2-4146-86c4-6e09bb663d60" containerID="584b939da10e238bb5679b9d7b78edf44ff5134caebb283e9d8652c8087c4464" exitCode=0
Jul 30 02:37:05.376351 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.376275    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"
Jul 30 02:37:05.376837 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:37:05.376808    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wqt8k\" is forbidden: User \"system:node:ip-10-0-132-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-89.ec2.internal' and this object" podUID="12569a05-86c2-4146-86c4-6e09bb663d60" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"
Jul 30 02:37:05.378875 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.378839    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zsg96" event={"ID":"f8fe9862-0773-4e89-b2b3-d4bd37967bbb","Type":"ContainerStarted","Data":"e1c8a37cbd2e6fae9061650a1b0a417c399ec23cb20a3d9aeccf146a114b483a"}
Jul 30 02:37:05.379031 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:37:05.379005    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z8ck5\" is forbidden: User \"system:node:ip-10-0-132-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-89.ec2.internal' and this object" podUID="fdbd7ae8-ae26-4213-bdf1-f75542880464" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"
Jul 30 02:37:05.380621 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.380597    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll" event={"ID":"79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8","Type":"ContainerStarted","Data":"19c8f16191418c26ec35229b1a7790ed25b6877a87c81f245c6580522afe3d83"}
Jul 30 02:37:05.380723 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.380647    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll" event={"ID":"79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8","Type":"ContainerStarted","Data":"55b02b5fb26b22d2d2cda3a6c8f8bc6205f304243442517373f666028a87eeaa"}
Jul 30 02:37:05.380988 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.380974    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll"
Jul 30 02:37:05.385433 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.385409    2565 scope.go:117] "RemoveContainer" containerID="6d9bec19578198eccea0bd22ca164f6470d40d06719b4dd882b9da0058e49522"
Jul 30 02:37:05.385707 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:37:05.385689    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d9bec19578198eccea0bd22ca164f6470d40d06719b4dd882b9da0058e49522\": container with ID starting with 6d9bec19578198eccea0bd22ca164f6470d40d06719b4dd882b9da0058e49522 not found: ID does not exist" containerID="6d9bec19578198eccea0bd22ca164f6470d40d06719b4dd882b9da0058e49522"
Jul 30 02:37:05.385774 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.385715    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9bec19578198eccea0bd22ca164f6470d40d06719b4dd882b9da0058e49522"} err="failed to get container status \"6d9bec19578198eccea0bd22ca164f6470d40d06719b4dd882b9da0058e49522\": rpc error: code = NotFound desc = could not find container \"6d9bec19578198eccea0bd22ca164f6470d40d06719b4dd882b9da0058e49522\": container with ID starting with 6d9bec19578198eccea0bd22ca164f6470d40d06719b4dd882b9da0058e49522 not found: ID does not exist"
Jul 30 02:37:05.385774 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.385735    2565 scope.go:117] "RemoveContainer" containerID="584b939da10e238bb5679b9d7b78edf44ff5134caebb283e9d8652c8087c4464"
Jul 30 02:37:05.393772 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.393755    2565 scope.go:117] "RemoveContainer" containerID="584b939da10e238bb5679b9d7b78edf44ff5134caebb283e9d8652c8087c4464"
Jul 30 02:37:05.394112 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:37:05.394059    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"584b939da10e238bb5679b9d7b78edf44ff5134caebb283e9d8652c8087c4464\": container with ID starting with 584b939da10e238bb5679b9d7b78edf44ff5134caebb283e9d8652c8087c4464 not found: ID does not exist" containerID="584b939da10e238bb5679b9d7b78edf44ff5134caebb283e9d8652c8087c4464"
Jul 30 02:37:05.394236 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.394120    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"584b939da10e238bb5679b9d7b78edf44ff5134caebb283e9d8652c8087c4464"} err="failed to get container status \"584b939da10e238bb5679b9d7b78edf44ff5134caebb283e9d8652c8087c4464\": rpc error: code = NotFound desc = could not find container \"584b939da10e238bb5679b9d7b78edf44ff5134caebb283e9d8652c8087c4464\": container with ID starting with 584b939da10e238bb5679b9d7b78edf44ff5134caebb283e9d8652c8087c4464 not found: ID does not exist"
Jul 30 02:37:05.396177 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:37:05.396140    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wqt8k\" is forbidden: User \"system:node:ip-10-0-132-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-89.ec2.internal' and this object" podUID="12569a05-86c2-4146-86c4-6e09bb663d60" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"
Jul 30 02:37:05.397119 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.397055    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zsg96" podStartSLOduration=2.232847478 podStartE2EDuration="23.397042418s" totalImagesPullingTime="21.16419494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:37:05.394124663 +0000 UTC m=+661.040908067" watchObservedRunningTime="2026-07-30 02:37:05.397042418 +0000 UTC m=+661.043825885"
Jul 30 02:37:05.398221 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:37:05.398202    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z8ck5\" is forbidden: User \"system:node:ip-10-0-132-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-89.ec2.internal' and this object" podUID="fdbd7ae8-ae26-4213-bdf1-f75542880464" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"
Jul 30 02:37:05.400330 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:37:05.400272    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wqt8k\" is forbidden: User \"system:node:ip-10-0-132-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-89.ec2.internal' and this object" podUID="12569a05-86c2-4146-86c4-6e09bb663d60" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wqt8k"
Jul 30 02:37:05.416597 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:05.416560    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll" podStartSLOduration=13.41654942 podStartE2EDuration="13.41654942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:37:05.416190522 +0000 UTC m=+661.062973926" watchObservedRunningTime="2026-07-30 02:37:05.41654942 +0000 UTC m=+661.063332825"
Jul 30 02:37:05.418288 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:37:05.418263    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z8ck5\" is forbidden: User \"system:node:ip-10-0-132-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-89.ec2.internal' and this object" podUID="fdbd7ae8-ae26-4213-bdf1-f75542880464" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z8ck5"
Jul 30 02:37:06.806473 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:06.806440    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12569a05-86c2-4146-86c4-6e09bb663d60" path="/var/lib/kubelet/pods/12569a05-86c2-4146-86c4-6e09bb663d60/volumes"
Jul 30 02:37:06.806843 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:06.806746    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdbd7ae8-ae26-4213-bdf1-f75542880464" path="/var/lib/kubelet/pods/fdbd7ae8-ae26-4213-bdf1-f75542880464/volumes"
Jul 30 02:37:09.199488 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.199453    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll"]
Jul 30 02:37:09.199868 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.199631    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll" podUID="79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8" containerName="manager" containerID="cri-o://19c8f16191418c26ec35229b1a7790ed25b6877a87c81f245c6580522afe3d83" gracePeriod=10
Jul 30 02:37:09.404703 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.404672    2565 generic.go:356] "Generic (PLEG): container finished" podID="79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8" containerID="19c8f16191418c26ec35229b1a7790ed25b6877a87c81f245c6580522afe3d83" exitCode=0
Jul 30 02:37:09.404860 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.404752    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll" event={"ID":"79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8","Type":"ContainerDied","Data":"19c8f16191418c26ec35229b1a7790ed25b6877a87c81f245c6580522afe3d83"}
Jul 30 02:37:09.442599 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.442575    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll"
Jul 30 02:37:09.513147 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.513070    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz"]
Jul 30 02:37:09.513438 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.513425    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8" containerName="manager"
Jul 30 02:37:09.513493 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.513440    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8" containerName="manager"
Jul 30 02:37:09.513493 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.513454    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdbd7ae8-ae26-4213-bdf1-f75542880464" containerName="manager"
Jul 30 02:37:09.513493 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.513460    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdbd7ae8-ae26-4213-bdf1-f75542880464" containerName="manager"
Jul 30 02:37:09.513595 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.513513    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdbd7ae8-ae26-4213-bdf1-f75542880464" containerName="manager"
Jul 30 02:37:09.513595 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.513525    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8" containerName="manager"
Jul 30 02:37:09.516920 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.516906    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz"
Jul 30 02:37:09.530019 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.529993    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz"]
Jul 30 02:37:09.588748 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.588723    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnztr\" (UniqueName: \"kubernetes.io/projected/79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8-kube-api-access-jnztr\") pod \"79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8\" (UID: \"79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8\") "
Jul 30 02:37:09.588887 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.588790    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8-extensions-socket-volume\") pod \"79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8\" (UID: \"79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8\") "
Jul 30 02:37:09.589217 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.589188    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8" (UID: "79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:37:09.590946 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.590923    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8-kube-api-access-jnztr" (OuterVolumeSpecName: "kube-api-access-jnztr") pod "79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8" (UID: "79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8"). InnerVolumeSpecName "kube-api-access-jnztr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:37:09.689642 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.689607    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/985a4301-e06d-4ab1-b967-8d2eaff835f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pkthz\" (UID: \"985a4301-e06d-4ab1-b967-8d2eaff835f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz"
Jul 30 02:37:09.689794 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.689648    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnxg5\" (UniqueName: \"kubernetes.io/projected/985a4301-e06d-4ab1-b967-8d2eaff835f0-kube-api-access-nnxg5\") pod \"kuadrant-operator-controller-manager-86878c769f-pkthz\" (UID: \"985a4301-e06d-4ab1-b967-8d2eaff835f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz"
Jul 30 02:37:09.689794 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.689737    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8-extensions-socket-volume\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:37:09.689794 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.689749    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnztr\" (UniqueName: \"kubernetes.io/projected/79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8-kube-api-access-jnztr\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:37:09.790444 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.790413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnxg5\" (UniqueName: \"kubernetes.io/projected/985a4301-e06d-4ab1-b967-8d2eaff835f0-kube-api-access-nnxg5\") pod \"kuadrant-operator-controller-manager-86878c769f-pkthz\" (UID: \"985a4301-e06d-4ab1-b967-8d2eaff835f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz"
Jul 30 02:37:09.790587 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.790500    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/985a4301-e06d-4ab1-b967-8d2eaff835f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pkthz\" (UID: \"985a4301-e06d-4ab1-b967-8d2eaff835f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz"
Jul 30 02:37:09.790809 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.790795    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/985a4301-e06d-4ab1-b967-8d2eaff835f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pkthz\" (UID: \"985a4301-e06d-4ab1-b967-8d2eaff835f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz"
Jul 30 02:37:09.803915 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.803889    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnxg5\" (UniqueName: \"kubernetes.io/projected/985a4301-e06d-4ab1-b967-8d2eaff835f0-kube-api-access-nnxg5\") pod \"kuadrant-operator-controller-manager-86878c769f-pkthz\" (UID: \"985a4301-e06d-4ab1-b967-8d2eaff835f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz"
Jul 30 02:37:09.826941 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.826905    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz"
Jul 30 02:37:09.986354 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:09.986326    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz"]
Jul 30 02:37:09.987472 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:37:09.987449    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985a4301_e06d_4ab1_b967_8d2eaff835f0.slice/crio-13dcb0556a95bd54a1ece5e8c2e2119d85249773b31bd03ec5b9af5175173dc2 WatchSource:0}: Error finding container 13dcb0556a95bd54a1ece5e8c2e2119d85249773b31bd03ec5b9af5175173dc2: Status 404 returned error can't find the container with id 13dcb0556a95bd54a1ece5e8c2e2119d85249773b31bd03ec5b9af5175173dc2
Jul 30 02:37:10.411126 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:10.411044    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll"
Jul 30 02:37:10.411126 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:10.411043    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll" event={"ID":"79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8","Type":"ContainerDied","Data":"55b02b5fb26b22d2d2cda3a6c8f8bc6205f304243442517373f666028a87eeaa"}
Jul 30 02:37:10.411635 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:10.411153    2565 scope.go:117] "RemoveContainer" containerID="19c8f16191418c26ec35229b1a7790ed25b6877a87c81f245c6580522afe3d83"
Jul 30 02:37:10.412942 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:10.412899    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz" event={"ID":"985a4301-e06d-4ab1-b967-8d2eaff835f0","Type":"ContainerStarted","Data":"b4877a0e384273257c0cbc4c6a0c24192c2315e4e0b3feae7add3258e349ef3f"}
Jul 30 02:37:10.413038 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:10.412967    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz"
Jul 30 02:37:10.413038 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:10.413026    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz" event={"ID":"985a4301-e06d-4ab1-b967-8d2eaff835f0","Type":"ContainerStarted","Data":"13dcb0556a95bd54a1ece5e8c2e2119d85249773b31bd03ec5b9af5175173dc2"}
Jul 30 02:37:10.440053 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:10.440012    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz" podStartSLOduration=1.439998041 podStartE2EDuration="1.439998041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:37:10.438940824 +0000 UTC m=+666.085724225" watchObservedRunningTime="2026-07-30 02:37:10.439998041 +0000 UTC m=+666.086781445"
Jul 30 02:37:10.457556 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:10.457532    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll"]
Jul 30 02:37:10.465536 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:10.465515    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zfbll"]
Jul 30 02:37:10.800125 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:10.800084    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8" path="/var/lib/kubelet/pods/79cf9fd9-3894-4fc3-a5cc-a8b04511e1e8/volumes"
Jul 30 02:37:21.427578 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:21.427545    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz"
Jul 30 02:37:25.560399 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.560365    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"]
Jul 30 02:37:25.763252 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.763209    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"]
Jul 30 02:37:25.763418 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.763266    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.766346 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.766324    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lv7l5\""
Jul 30 02:37:25.826492 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.826405    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd921d07-bc19-45b2-b724-43dce72b1d84-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.826492 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.826460    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd921d07-bc19-45b2-b724-43dce72b1d84-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.826492 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.826487    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd921d07-bc19-45b2-b724-43dce72b1d84-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.826711 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.826517    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd921d07-bc19-45b2-b724-43dce72b1d84-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.826711 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.826547    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd921d07-bc19-45b2-b724-43dce72b1d84-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.826711 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.826598    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69rmb\" (UniqueName: \"kubernetes.io/projected/cd921d07-bc19-45b2-b724-43dce72b1d84-kube-api-access-69rmb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.826711 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.826639    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd921d07-bc19-45b2-b724-43dce72b1d84-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.826711 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.826669    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd921d07-bc19-45b2-b724-43dce72b1d84-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.826872 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.826731    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd921d07-bc19-45b2-b724-43dce72b1d84-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.927126 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.927086    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd921d07-bc19-45b2-b724-43dce72b1d84-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.927126 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.927130    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd921d07-bc19-45b2-b724-43dce72b1d84-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.927400 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.927163    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd921d07-bc19-45b2-b724-43dce72b1d84-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.927400 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.927222    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd921d07-bc19-45b2-b724-43dce72b1d84-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.927400 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.927255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69rmb\" (UniqueName: \"kubernetes.io/projected/cd921d07-bc19-45b2-b724-43dce72b1d84-kube-api-access-69rmb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.927400 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.927292    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd921d07-bc19-45b2-b724-43dce72b1d84-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.927400 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.927330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd921d07-bc19-45b2-b724-43dce72b1d84-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.927400 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.927382    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd921d07-bc19-45b2-b724-43dce72b1d84-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.927681 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.927430    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd921d07-bc19-45b2-b724-43dce72b1d84-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.927681 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.927577    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd921d07-bc19-45b2-b724-43dce72b1d84-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.927932 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.927882    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd921d07-bc19-45b2-b724-43dce72b1d84-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.927932 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.927860    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd921d07-bc19-45b2-b724-43dce72b1d84-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.928100 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.927983    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd921d07-bc19-45b2-b724-43dce72b1d84-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.928100 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.927922    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd921d07-bc19-45b2-b724-43dce72b1d84-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.929689 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.929666    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd921d07-bc19-45b2-b724-43dce72b1d84-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.929895 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.929876    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd921d07-bc19-45b2-b724-43dce72b1d84-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.942686 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.942654    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd921d07-bc19-45b2-b724-43dce72b1d84-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:25.942956 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:25.942939    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69rmb\" (UniqueName: \"kubernetes.io/projected/cd921d07-bc19-45b2-b724-43dce72b1d84-kube-api-access-69rmb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9vf5j\" (UID: \"cd921d07-bc19-45b2-b724-43dce72b1d84\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:26.073662 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:26.073624    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:26.238946 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:26.238919    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"]
Jul 30 02:37:26.240390 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:37:26.240348    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd921d07_bc19_45b2_b724_43dce72b1d84.slice/crio-61a472e59b13395bf6de97f095393b865ae83d3c1e14f5d73622f88e0895238f WatchSource:0}: Error finding container 61a472e59b13395bf6de97f095393b865ae83d3c1e14f5d73622f88e0895238f: Status 404 returned error can't find the container with id 61a472e59b13395bf6de97f095393b865ae83d3c1e14f5d73622f88e0895238f
Jul 30 02:37:26.242896 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:26.242860    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:37:26.243006 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:26.242928    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:37:26.243006 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:26.242956    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:37:26.507984 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:26.507947    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j" event={"ID":"cd921d07-bc19-45b2-b724-43dce72b1d84","Type":"ContainerStarted","Data":"9f928dacae49f09ba1a19eb19030570cfb1a8b41556a2d9d9088f7e1946a9fbc"}
Jul 30 02:37:26.508203 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:26.508011    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j" event={"ID":"cd921d07-bc19-45b2-b724-43dce72b1d84","Type":"ContainerStarted","Data":"61a472e59b13395bf6de97f095393b865ae83d3c1e14f5d73622f88e0895238f"}
Jul 30 02:37:26.529479 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:26.529426    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j" podStartSLOduration=1.52940996 podStartE2EDuration="1.52940996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:37:26.527022887 +0000 UTC m=+682.173806298" watchObservedRunningTime="2026-07-30 02:37:26.52940996 +0000 UTC m=+682.176193366"
Jul 30 02:37:27.074499 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:27.074464    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:27.079206 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:27.079158    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:27.513888 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:27.513811    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:27.514774 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:27.514759    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:37:39.712836 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:39.712804    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-l8nxs"]
Jul 30 02:37:39.779992 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:39.779938    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-l8nxs"]
Jul 30 02:37:39.780468 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:39.780143    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-l8nxs"
Jul 30 02:37:39.783518 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:39.783497    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 30 02:37:39.820704 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:39.820668    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-l8nxs"]
Jul 30 02:37:39.846327 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:39.846295    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vpj2\" (UniqueName: \"kubernetes.io/projected/a887998b-4821-4a50-a762-e62f459d62d3-kube-api-access-4vpj2\") pod \"limitador-limitador-64c94fd64f-l8nxs\" (UID: \"a887998b-4821-4a50-a762-e62f459d62d3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-l8nxs"
Jul 30 02:37:39.846489 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:39.846359    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a887998b-4821-4a50-a762-e62f459d62d3-config-file\") pod \"limitador-limitador-64c94fd64f-l8nxs\" (UID: \"a887998b-4821-4a50-a762-e62f459d62d3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-l8nxs"
Jul 30 02:37:39.947580 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:39.947536    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a887998b-4821-4a50-a762-e62f459d62d3-config-file\") pod \"limitador-limitador-64c94fd64f-l8nxs\" (UID: \"a887998b-4821-4a50-a762-e62f459d62d3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-l8nxs"
Jul 30 02:37:39.947775 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:39.947652    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vpj2\" (UniqueName: \"kubernetes.io/projected/a887998b-4821-4a50-a762-e62f459d62d3-kube-api-access-4vpj2\") pod \"limitador-limitador-64c94fd64f-l8nxs\" (UID: \"a887998b-4821-4a50-a762-e62f459d62d3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-l8nxs"
Jul 30 02:37:39.948211 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:39.948188    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a887998b-4821-4a50-a762-e62f459d62d3-config-file\") pod \"limitador-limitador-64c94fd64f-l8nxs\" (UID: \"a887998b-4821-4a50-a762-e62f459d62d3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-l8nxs"
Jul 30 02:37:39.956253 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:39.956226    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vpj2\" (UniqueName: \"kubernetes.io/projected/a887998b-4821-4a50-a762-e62f459d62d3-kube-api-access-4vpj2\") pod \"limitador-limitador-64c94fd64f-l8nxs\" (UID: \"a887998b-4821-4a50-a762-e62f459d62d3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-l8nxs"
Jul 30 02:37:40.093589 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:40.093552    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-l8nxs"
Jul 30 02:37:40.230600 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:40.230573    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-l8nxs"]
Jul 30 02:37:40.232688 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:37:40.232663    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda887998b_4821_4a50_a762_e62f459d62d3.slice/crio-97a0e52e062dfdc488ddb13738baf57f37422c2d759b929923b06f7ccf7bc160 WatchSource:0}: Error finding container 97a0e52e062dfdc488ddb13738baf57f37422c2d759b929923b06f7ccf7bc160: Status 404 returned error can't find the container with id 97a0e52e062dfdc488ddb13738baf57f37422c2d759b929923b06f7ccf7bc160
Jul 30 02:37:40.234497 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:40.234479    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 02:37:40.527268 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:40.527182    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-rr79x"]
Jul 30 02:37:40.532450 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:40.532422    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rr79x"
Jul 30 02:37:40.535694 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:40.535392    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kg85n\""
Jul 30 02:37:40.539240 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:40.539215    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rr79x"]
Jul 30 02:37:40.583808 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:40.583775    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-l8nxs" event={"ID":"a887998b-4821-4a50-a762-e62f459d62d3","Type":"ContainerStarted","Data":"97a0e52e062dfdc488ddb13738baf57f37422c2d759b929923b06f7ccf7bc160"}
Jul 30 02:37:40.653190 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:40.653131    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpgzb\" (UniqueName: \"kubernetes.io/projected/b9aa4552-225f-4c52-bde3-fd0e2839a194-kube-api-access-wpgzb\") pod \"authorino-58cd955675-rr79x\" (UID: \"b9aa4552-225f-4c52-bde3-fd0e2839a194\") " pod="kuadrant-system/authorino-58cd955675-rr79x"
Jul 30 02:37:40.753903 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:40.753869    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpgzb\" (UniqueName: \"kubernetes.io/projected/b9aa4552-225f-4c52-bde3-fd0e2839a194-kube-api-access-wpgzb\") pod \"authorino-58cd955675-rr79x\" (UID: \"b9aa4552-225f-4c52-bde3-fd0e2839a194\") " pod="kuadrant-system/authorino-58cd955675-rr79x"
Jul 30 02:37:40.776452 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:40.776422    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpgzb\" (UniqueName: \"kubernetes.io/projected/b9aa4552-225f-4c52-bde3-fd0e2839a194-kube-api-access-wpgzb\") pod \"authorino-58cd955675-rr79x\" (UID: \"b9aa4552-225f-4c52-bde3-fd0e2839a194\") " pod="kuadrant-system/authorino-58cd955675-rr79x"
Jul 30 02:37:40.843693 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:40.843660    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rr79x"
Jul 30 02:37:41.010985 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:41.010947    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rr79x"]
Jul 30 02:37:41.015749 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:37:41.015716    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9aa4552_225f_4c52_bde3_fd0e2839a194.slice/crio-1f606ac422fde5c4492247f29c6bacd5aa96659d45ef2c90286035bca199bd72 WatchSource:0}: Error finding container 1f606ac422fde5c4492247f29c6bacd5aa96659d45ef2c90286035bca199bd72: Status 404 returned error can't find the container with id 1f606ac422fde5c4492247f29c6bacd5aa96659d45ef2c90286035bca199bd72
Jul 30 02:37:41.592367 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:41.592324    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rr79x" event={"ID":"b9aa4552-225f-4c52-bde3-fd0e2839a194","Type":"ContainerStarted","Data":"1f606ac422fde5c4492247f29c6bacd5aa96659d45ef2c90286035bca199bd72"}
Jul 30 02:37:45.408853 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:45.408820    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rr79x"]
Jul 30 02:37:45.618797 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:45.618762    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-l8nxs" event={"ID":"a887998b-4821-4a50-a762-e62f459d62d3","Type":"ContainerStarted","Data":"2b84678ca95721cdd4303246c458bd7eba0f2300c9d70d759eb38fa8bd1fb648"}
Jul 30 02:37:45.618983 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:45.618927    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-l8nxs"
Jul 30 02:37:45.620405 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:45.620379    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rr79x" event={"ID":"b9aa4552-225f-4c52-bde3-fd0e2839a194","Type":"ContainerStarted","Data":"08cdcca7a7ec496d52a5db769fc0e911855f02c8dd544fae19b6c2a868c62992"}
Jul 30 02:37:45.636151 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:45.636102    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-l8nxs" podStartSLOduration=2.051622938 podStartE2EDuration="6.636087884s" totalImagesPullingTime="4.584464946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:37:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:37:45.634153774 +0000 UTC m=+701.280937181" watchObservedRunningTime="2026-07-30 02:37:45.636087884 +0000 UTC m=+701.282871289"
Jul 30 02:37:45.649373 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:45.649329    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-rr79x" podStartSLOduration=1.836946728 podStartE2EDuration="5.649315272s" totalImagesPullingTime="3.812368544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:37:45.647425416 +0000 UTC m=+701.294208834" watchObservedRunningTime="2026-07-30 02:37:45.649315272 +0000 UTC m=+701.296098677"
Jul 30 02:37:46.625889 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:46.625817    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-rr79x" podUID="b9aa4552-225f-4c52-bde3-fd0e2839a194" containerName="authorino" containerID="cri-o://08cdcca7a7ec496d52a5db769fc0e911855f02c8dd544fae19b6c2a868c62992" gracePeriod=30
Jul 30 02:37:46.879533 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:46.879477    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rr79x"
Jul 30 02:37:47.011417 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:47.011386    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpgzb\" (UniqueName: \"kubernetes.io/projected/b9aa4552-225f-4c52-bde3-fd0e2839a194-kube-api-access-wpgzb\") pod \"b9aa4552-225f-4c52-bde3-fd0e2839a194\" (UID: \"b9aa4552-225f-4c52-bde3-fd0e2839a194\") "
Jul 30 02:37:47.013584 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:47.013555    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9aa4552-225f-4c52-bde3-fd0e2839a194-kube-api-access-wpgzb" (OuterVolumeSpecName: "kube-api-access-wpgzb") pod "b9aa4552-225f-4c52-bde3-fd0e2839a194" (UID: "b9aa4552-225f-4c52-bde3-fd0e2839a194"). InnerVolumeSpecName "kube-api-access-wpgzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:37:47.112483 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:47.112451    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpgzb\" (UniqueName: \"kubernetes.io/projected/b9aa4552-225f-4c52-bde3-fd0e2839a194-kube-api-access-wpgzb\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:37:47.632562 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:47.632526    2565 generic.go:356] "Generic (PLEG): container finished" podID="b9aa4552-225f-4c52-bde3-fd0e2839a194" containerID="08cdcca7a7ec496d52a5db769fc0e911855f02c8dd544fae19b6c2a868c62992" exitCode=0
Jul 30 02:37:47.633031 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:47.632574    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rr79x"
Jul 30 02:37:47.633031 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:47.632611    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rr79x" event={"ID":"b9aa4552-225f-4c52-bde3-fd0e2839a194","Type":"ContainerDied","Data":"08cdcca7a7ec496d52a5db769fc0e911855f02c8dd544fae19b6c2a868c62992"}
Jul 30 02:37:47.633031 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:47.632652    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rr79x" event={"ID":"b9aa4552-225f-4c52-bde3-fd0e2839a194","Type":"ContainerDied","Data":"1f606ac422fde5c4492247f29c6bacd5aa96659d45ef2c90286035bca199bd72"}
Jul 30 02:37:47.633031 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:47.632671    2565 scope.go:117] "RemoveContainer" containerID="08cdcca7a7ec496d52a5db769fc0e911855f02c8dd544fae19b6c2a868c62992"
Jul 30 02:37:47.641868 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:47.641850    2565 scope.go:117] "RemoveContainer" containerID="08cdcca7a7ec496d52a5db769fc0e911855f02c8dd544fae19b6c2a868c62992"
Jul 30 02:37:47.642135 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:37:47.642116    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08cdcca7a7ec496d52a5db769fc0e911855f02c8dd544fae19b6c2a868c62992\": container with ID starting with 08cdcca7a7ec496d52a5db769fc0e911855f02c8dd544fae19b6c2a868c62992 not found: ID does not exist" containerID="08cdcca7a7ec496d52a5db769fc0e911855f02c8dd544fae19b6c2a868c62992"
Jul 30 02:37:47.642219 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:47.642145    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08cdcca7a7ec496d52a5db769fc0e911855f02c8dd544fae19b6c2a868c62992"} err="failed to get container status \"08cdcca7a7ec496d52a5db769fc0e911855f02c8dd544fae19b6c2a868c62992\": rpc error: code = NotFound desc = could not find container \"08cdcca7a7ec496d52a5db769fc0e911855f02c8dd544fae19b6c2a868c62992\": container with ID starting with 08cdcca7a7ec496d52a5db769fc0e911855f02c8dd544fae19b6c2a868c62992 not found: ID does not exist"
Jul 30 02:37:47.653678 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:47.653650    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rr79x"]
Jul 30 02:37:47.659747 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:47.659723    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-rr79x"]
Jul 30 02:37:48.800424 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:48.800386    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9aa4552-225f-4c52-bde3-fd0e2839a194" path="/var/lib/kubelet/pods/b9aa4552-225f-4c52-bde3-fd0e2839a194/volumes"
Jul 30 02:37:54.403860 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:54.403824    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-l8nxs"]
Jul 30 02:37:54.404327 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:54.404018    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-l8nxs" podUID="a887998b-4821-4a50-a762-e62f459d62d3" containerName="limitador" containerID="cri-o://2b84678ca95721cdd4303246c458bd7eba0f2300c9d70d759eb38fa8bd1fb648" gracePeriod=30
Jul 30 02:37:54.404749 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:54.404729    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-l8nxs"
Jul 30 02:37:54.962045 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:54.962022    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-l8nxs"
Jul 30 02:37:55.084735 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:55.084699    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a887998b-4821-4a50-a762-e62f459d62d3-config-file\") pod \"a887998b-4821-4a50-a762-e62f459d62d3\" (UID: \"a887998b-4821-4a50-a762-e62f459d62d3\") "
Jul 30 02:37:55.084942 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:55.084760    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vpj2\" (UniqueName: \"kubernetes.io/projected/a887998b-4821-4a50-a762-e62f459d62d3-kube-api-access-4vpj2\") pod \"a887998b-4821-4a50-a762-e62f459d62d3\" (UID: \"a887998b-4821-4a50-a762-e62f459d62d3\") "
Jul 30 02:37:55.085078 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:55.085051    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a887998b-4821-4a50-a762-e62f459d62d3-config-file" (OuterVolumeSpecName: "config-file") pod "a887998b-4821-4a50-a762-e62f459d62d3" (UID: "a887998b-4821-4a50-a762-e62f459d62d3"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 02:37:55.087241 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:55.087204    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a887998b-4821-4a50-a762-e62f459d62d3-kube-api-access-4vpj2" (OuterVolumeSpecName: "kube-api-access-4vpj2") pod "a887998b-4821-4a50-a762-e62f459d62d3" (UID: "a887998b-4821-4a50-a762-e62f459d62d3"). InnerVolumeSpecName "kube-api-access-4vpj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:37:55.186098 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:55.186056    2565 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a887998b-4821-4a50-a762-e62f459d62d3-config-file\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:37:55.186098 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:55.186090    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vpj2\" (UniqueName: \"kubernetes.io/projected/a887998b-4821-4a50-a762-e62f459d62d3-kube-api-access-4vpj2\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:37:55.675409 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:55.675331    2565 generic.go:356] "Generic (PLEG): container finished" podID="a887998b-4821-4a50-a762-e62f459d62d3" containerID="2b84678ca95721cdd4303246c458bd7eba0f2300c9d70d759eb38fa8bd1fb648" exitCode=0
Jul 30 02:37:55.675409 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:55.675380    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-l8nxs"
Jul 30 02:37:55.675817 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:55.675378    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-l8nxs" event={"ID":"a887998b-4821-4a50-a762-e62f459d62d3","Type":"ContainerDied","Data":"2b84678ca95721cdd4303246c458bd7eba0f2300c9d70d759eb38fa8bd1fb648"}
Jul 30 02:37:55.675817 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:55.675488    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-l8nxs" event={"ID":"a887998b-4821-4a50-a762-e62f459d62d3","Type":"ContainerDied","Data":"97a0e52e062dfdc488ddb13738baf57f37422c2d759b929923b06f7ccf7bc160"}
Jul 30 02:37:55.675817 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:55.675514    2565 scope.go:117] "RemoveContainer" containerID="2b84678ca95721cdd4303246c458bd7eba0f2300c9d70d759eb38fa8bd1fb648"
Jul 30 02:37:55.684265 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:55.684245    2565 scope.go:117] "RemoveContainer" containerID="2b84678ca95721cdd4303246c458bd7eba0f2300c9d70d759eb38fa8bd1fb648"
Jul 30 02:37:55.684527 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:37:55.684506    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b84678ca95721cdd4303246c458bd7eba0f2300c9d70d759eb38fa8bd1fb648\": container with ID starting with 2b84678ca95721cdd4303246c458bd7eba0f2300c9d70d759eb38fa8bd1fb648 not found: ID does not exist" containerID="2b84678ca95721cdd4303246c458bd7eba0f2300c9d70d759eb38fa8bd1fb648"
Jul 30 02:37:55.684590 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:55.684540    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b84678ca95721cdd4303246c458bd7eba0f2300c9d70d759eb38fa8bd1fb648"} err="failed to get container status \"2b84678ca95721cdd4303246c458bd7eba0f2300c9d70d759eb38fa8bd1fb648\": rpc error: code = NotFound desc = could not find container \"2b84678ca95721cdd4303246c458bd7eba0f2300c9d70d759eb38fa8bd1fb648\": container with ID starting with 2b84678ca95721cdd4303246c458bd7eba0f2300c9d70d759eb38fa8bd1fb648 not found: ID does not exist"
Jul 30 02:37:55.697584 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:55.697557    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-l8nxs"]
Jul 30 02:37:55.700944 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:55.700927    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-l8nxs"]
Jul 30 02:37:56.800542 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:56.800507    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a887998b-4821-4a50-a762-e62f459d62d3" path="/var/lib/kubelet/pods/a887998b-4821-4a50-a762-e62f459d62d3/volumes"
Jul 30 02:37:59.368984 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.368946    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jkfmb"]
Jul 30 02:37:59.369443 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.369357    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a887998b-4821-4a50-a762-e62f459d62d3" containerName="limitador"
Jul 30 02:37:59.369443 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.369373    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a887998b-4821-4a50-a762-e62f459d62d3" containerName="limitador"
Jul 30 02:37:59.369443 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.369384    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9aa4552-225f-4c52-bde3-fd0e2839a194" containerName="authorino"
Jul 30 02:37:59.369443 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.369389    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9aa4552-225f-4c52-bde3-fd0e2839a194" containerName="authorino"
Jul 30 02:37:59.369615 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.369449    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a887998b-4821-4a50-a762-e62f459d62d3" containerName="limitador"
Jul 30 02:37:59.369615 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.369457    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9aa4552-225f-4c52-bde3-fd0e2839a194" containerName="authorino"
Jul 30 02:37:59.371378 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.371357    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jkfmb"
Jul 30 02:37:59.374137 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.374113    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 30 02:37:59.374137 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.374133    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 30 02:37:59.374308 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.374148    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 30 02:37:59.375488 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.375465    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-nv265\""
Jul 30 02:37:59.379904 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.379886    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jkfmb"]
Jul 30 02:37:59.527138 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.527102    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e03462aa-5f71-492f-92f4-7971c5ead0a8-data\") pod \"postgres-69fb4594cf-jkfmb\" (UID: \"e03462aa-5f71-492f-92f4-7971c5ead0a8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jkfmb"
Jul 30 02:37:59.527341 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.527151    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dff2d\" (UniqueName: \"kubernetes.io/projected/e03462aa-5f71-492f-92f4-7971c5ead0a8-kube-api-access-dff2d\") pod \"postgres-69fb4594cf-jkfmb\" (UID: \"e03462aa-5f71-492f-92f4-7971c5ead0a8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jkfmb"
Jul 30 02:37:59.627929 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.627836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e03462aa-5f71-492f-92f4-7971c5ead0a8-data\") pod \"postgres-69fb4594cf-jkfmb\" (UID: \"e03462aa-5f71-492f-92f4-7971c5ead0a8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jkfmb"
Jul 30 02:37:59.627929 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.627892    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dff2d\" (UniqueName: \"kubernetes.io/projected/e03462aa-5f71-492f-92f4-7971c5ead0a8-kube-api-access-dff2d\") pod \"postgres-69fb4594cf-jkfmb\" (UID: \"e03462aa-5f71-492f-92f4-7971c5ead0a8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jkfmb"
Jul 30 02:37:59.628274 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.628248    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e03462aa-5f71-492f-92f4-7971c5ead0a8-data\") pod \"postgres-69fb4594cf-jkfmb\" (UID: \"e03462aa-5f71-492f-92f4-7971c5ead0a8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jkfmb"
Jul 30 02:37:59.639093 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.639062    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dff2d\" (UniqueName: \"kubernetes.io/projected/e03462aa-5f71-492f-92f4-7971c5ead0a8-kube-api-access-dff2d\") pod \"postgres-69fb4594cf-jkfmb\" (UID: \"e03462aa-5f71-492f-92f4-7971c5ead0a8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jkfmb"
Jul 30 02:37:59.683069 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.683043    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jkfmb"
Jul 30 02:37:59.816390 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:37:59.816361    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jkfmb"]
Jul 30 02:37:59.818227 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:37:59.818157    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03462aa_5f71_492f_92f4_7971c5ead0a8.slice/crio-729bb83c90b09cbbefa10044e9f618640f80b386af22871ff97748cc7f5b3955 WatchSource:0}: Error finding container 729bb83c90b09cbbefa10044e9f618640f80b386af22871ff97748cc7f5b3955: Status 404 returned error can't find the container with id 729bb83c90b09cbbefa10044e9f618640f80b386af22871ff97748cc7f5b3955
Jul 30 02:38:00.704112 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:00.704041    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jkfmb" event={"ID":"e03462aa-5f71-492f-92f4-7971c5ead0a8","Type":"ContainerStarted","Data":"729bb83c90b09cbbefa10044e9f618640f80b386af22871ff97748cc7f5b3955"}
Jul 30 02:38:04.730397 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:04.730366    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jkfmb" event={"ID":"e03462aa-5f71-492f-92f4-7971c5ead0a8","Type":"ContainerStarted","Data":"d22404805b49c76beadaca67ce2841a42b0d51fcd89b8fda3a7af526036b5428"}
Jul 30 02:38:04.730717 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:04.730541    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jkfmb"
Jul 30 02:38:04.749044 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:04.749003    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jkfmb" podStartSLOduration=0.935016528 podStartE2EDuration="5.748987239s" totalImagesPullingTime="4.813970711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:38:04.747564862 +0000 UTC m=+720.394348269" watchObservedRunningTime="2026-07-30 02:38:04.748987239 +0000 UTC m=+720.395770643"
Jul 30 02:38:10.763698 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:10.763669    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jkfmb"
Jul 30 02:38:11.574240 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.574209    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-769fc7989b-f97wb"]
Jul 30 02:38:11.580283 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.580257    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-769fc7989b-f97wb"
Jul 30 02:38:11.583212 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.583162    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kg85n\""
Jul 30 02:38:11.584893 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.584869    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-769fc7989b-f97wb"]
Jul 30 02:38:11.642180 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.642142    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bxhq\" (UniqueName: \"kubernetes.io/projected/b7f3e169-7ca8-4c97-bdd1-7bb327b8d383-kube-api-access-2bxhq\") pod \"authorino-769fc7989b-f97wb\" (UID: \"b7f3e169-7ca8-4c97-bdd1-7bb327b8d383\") " pod="kuadrant-system/authorino-769fc7989b-f97wb"
Jul 30 02:38:11.673944 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.673911    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-769fc7989b-f97wb"]
Jul 30 02:38:11.674213 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:38:11.674187    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2bxhq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-769fc7989b-f97wb" podUID="b7f3e169-7ca8-4c97-bdd1-7bb327b8d383"
Jul 30 02:38:11.699978 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.699935    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5bd6b597bd-ww9hj"]
Jul 30 02:38:11.704133 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.704112    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bd6b597bd-ww9hj"
Jul 30 02:38:11.706911 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.706885    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 30 02:38:11.710559 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.710536    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5bd6b597bd-ww9hj"]
Jul 30 02:38:11.742742 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.742705    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d637db25-c915-479d-a504-6e1234ef4783-tls-cert\") pod \"authorino-5bd6b597bd-ww9hj\" (UID: \"d637db25-c915-479d-a504-6e1234ef4783\") " pod="kuadrant-system/authorino-5bd6b597bd-ww9hj"
Jul 30 02:38:11.742894 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.742808    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szbgh\" (UniqueName: \"kubernetes.io/projected/d637db25-c915-479d-a504-6e1234ef4783-kube-api-access-szbgh\") pod \"authorino-5bd6b597bd-ww9hj\" (UID: \"d637db25-c915-479d-a504-6e1234ef4783\") " pod="kuadrant-system/authorino-5bd6b597bd-ww9hj"
Jul 30 02:38:11.742894 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.742852    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bxhq\" (UniqueName: \"kubernetes.io/projected/b7f3e169-7ca8-4c97-bdd1-7bb327b8d383-kube-api-access-2bxhq\") pod \"authorino-769fc7989b-f97wb\" (UID: \"b7f3e169-7ca8-4c97-bdd1-7bb327b8d383\") " pod="kuadrant-system/authorino-769fc7989b-f97wb"
Jul 30 02:38:11.753598 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.753571    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bxhq\" (UniqueName: \"kubernetes.io/projected/b7f3e169-7ca8-4c97-bdd1-7bb327b8d383-kube-api-access-2bxhq\") pod \"authorino-769fc7989b-f97wb\" (UID: \"b7f3e169-7ca8-4c97-bdd1-7bb327b8d383\") " pod="kuadrant-system/authorino-769fc7989b-f97wb"
Jul 30 02:38:11.771812 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.771787    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-769fc7989b-f97wb"
Jul 30 02:38:11.779192 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.779154    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-769fc7989b-f97wb"
Jul 30 02:38:11.843396 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.843295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d637db25-c915-479d-a504-6e1234ef4783-tls-cert\") pod \"authorino-5bd6b597bd-ww9hj\" (UID: \"d637db25-c915-479d-a504-6e1234ef4783\") " pod="kuadrant-system/authorino-5bd6b597bd-ww9hj"
Jul 30 02:38:11.843396 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.843381    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szbgh\" (UniqueName: \"kubernetes.io/projected/d637db25-c915-479d-a504-6e1234ef4783-kube-api-access-szbgh\") pod \"authorino-5bd6b597bd-ww9hj\" (UID: \"d637db25-c915-479d-a504-6e1234ef4783\") " pod="kuadrant-system/authorino-5bd6b597bd-ww9hj"
Jul 30 02:38:11.846024 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.845999    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d637db25-c915-479d-a504-6e1234ef4783-tls-cert\") pod \"authorino-5bd6b597bd-ww9hj\" (UID: \"d637db25-c915-479d-a504-6e1234ef4783\") " pod="kuadrant-system/authorino-5bd6b597bd-ww9hj"
Jul 30 02:38:11.852960 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.852931    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szbgh\" (UniqueName: \"kubernetes.io/projected/d637db25-c915-479d-a504-6e1234ef4783-kube-api-access-szbgh\") pod \"authorino-5bd6b597bd-ww9hj\" (UID: \"d637db25-c915-479d-a504-6e1234ef4783\") " pod="kuadrant-system/authorino-5bd6b597bd-ww9hj"
Jul 30 02:38:11.944693 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.944659    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bxhq\" (UniqueName: \"kubernetes.io/projected/b7f3e169-7ca8-4c97-bdd1-7bb327b8d383-kube-api-access-2bxhq\") pod \"b7f3e169-7ca8-4c97-bdd1-7bb327b8d383\" (UID: \"b7f3e169-7ca8-4c97-bdd1-7bb327b8d383\") "
Jul 30 02:38:11.946790 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:11.946764    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f3e169-7ca8-4c97-bdd1-7bb327b8d383-kube-api-access-2bxhq" (OuterVolumeSpecName: "kube-api-access-2bxhq") pod "b7f3e169-7ca8-4c97-bdd1-7bb327b8d383" (UID: "b7f3e169-7ca8-4c97-bdd1-7bb327b8d383"). InnerVolumeSpecName "kube-api-access-2bxhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:38:12.014004 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:12.013966    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bd6b597bd-ww9hj"
Jul 30 02:38:12.045845 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:12.045813    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bxhq\" (UniqueName: \"kubernetes.io/projected/b7f3e169-7ca8-4c97-bdd1-7bb327b8d383-kube-api-access-2bxhq\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:38:12.145396 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:12.145370    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5bd6b597bd-ww9hj"]
Jul 30 02:38:12.146994 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:38:12.146965    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd637db25_c915_479d_a504_6e1234ef4783.slice/crio-79f11794111bada36564d1e934bd057a9deaa06263c7d232613e84f8db78bd01 WatchSource:0}: Error finding container 79f11794111bada36564d1e934bd057a9deaa06263c7d232613e84f8db78bd01: Status 404 returned error can't find the container with id 79f11794111bada36564d1e934bd057a9deaa06263c7d232613e84f8db78bd01
Jul 30 02:38:12.778117 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:12.778031    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bd6b597bd-ww9hj" event={"ID":"d637db25-c915-479d-a504-6e1234ef4783","Type":"ContainerStarted","Data":"3709313825a600e2ce57e841d51942756b13b6cb5daf28a45ad3f7e1962257d6"}
Jul 30 02:38:12.778117 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:12.778072    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bd6b597bd-ww9hj" event={"ID":"d637db25-c915-479d-a504-6e1234ef4783","Type":"ContainerStarted","Data":"79f11794111bada36564d1e934bd057a9deaa06263c7d232613e84f8db78bd01"}
Jul 30 02:38:12.778117 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:12.778073    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-769fc7989b-f97wb"
Jul 30 02:38:12.801028 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:12.800985    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5bd6b597bd-ww9hj" podStartSLOduration=1.481224836 podStartE2EDuration="1.800971752s" totalImagesPullingTime="319.746916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:38:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:38:12.800236348 +0000 UTC m=+728.447019753" watchObservedRunningTime="2026-07-30 02:38:12.800971752 +0000 UTC m=+728.447755158"
Jul 30 02:38:12.825113 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:12.825077    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-769fc7989b-f97wb"]
Jul 30 02:38:12.835610 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:12.835578    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-769fc7989b-f97wb"]
Jul 30 02:38:14.800248 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:14.800215    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7f3e169-7ca8-4c97-bdd1-7bb327b8d383" path="/var/lib/kubelet/pods/b7f3e169-7ca8-4c97-bdd1-7bb327b8d383/volumes"
Jul 30 02:38:17.534345 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.534310    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d844fcccb-h5ckv"]
Jul 30 02:38:17.597105 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.597064    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d844fcccb-h5ckv"]
Jul 30 02:38:17.597293 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.597222    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d844fcccb-h5ckv"
Jul 30 02:38:17.601829 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.601796    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 30 02:38:17.601829 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.601815    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wxw25\""
Jul 30 02:38:17.602032 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.601816    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 30 02:38:17.660163 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.660130    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6899b54cb8-c9l7w"]
Jul 30 02:38:17.685077 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.685049    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6899b54cb8-c9l7w"]
Jul 30 02:38:17.685276 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.685244    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6899b54cb8-c9l7w"
Jul 30 02:38:17.687608 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.687586    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/620df212-4dc4-4298-9551-bbc5b71d84d2-webhook-cert\") pod \"maas-controller-6899b54cb8-c9l7w\" (UID: \"620df212-4dc4-4298-9551-bbc5b71d84d2\") " pod="opendatahub/maas-controller-6899b54cb8-c9l7w"
Jul 30 02:38:17.687772 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.687641    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/457bab40-a577-4197-8a29-9711cc6cff5c-webhook-cert\") pod \"maas-controller-6d844fcccb-h5ckv\" (UID: \"457bab40-a577-4197-8a29-9711cc6cff5c\") " pod="opendatahub/maas-controller-6d844fcccb-h5ckv"
Jul 30 02:38:17.687772 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.687688    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcphb\" (UniqueName: \"kubernetes.io/projected/457bab40-a577-4197-8a29-9711cc6cff5c-kube-api-access-tcphb\") pod \"maas-controller-6d844fcccb-h5ckv\" (UID: \"457bab40-a577-4197-8a29-9711cc6cff5c\") " pod="opendatahub/maas-controller-6d844fcccb-h5ckv"
Jul 30 02:38:17.687772 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.687728    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptvgv\" (UniqueName: \"kubernetes.io/projected/620df212-4dc4-4298-9551-bbc5b71d84d2-kube-api-access-ptvgv\") pod \"maas-controller-6899b54cb8-c9l7w\" (UID: \"620df212-4dc4-4298-9551-bbc5b71d84d2\") " pod="opendatahub/maas-controller-6899b54cb8-c9l7w"
Jul 30 02:38:17.788885 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.788858    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/620df212-4dc4-4298-9551-bbc5b71d84d2-webhook-cert\") pod \"maas-controller-6899b54cb8-c9l7w\" (UID: \"620df212-4dc4-4298-9551-bbc5b71d84d2\") " pod="opendatahub/maas-controller-6899b54cb8-c9l7w"
Jul 30 02:38:17.789079 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.788929    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/457bab40-a577-4197-8a29-9711cc6cff5c-webhook-cert\") pod \"maas-controller-6d844fcccb-h5ckv\" (UID: \"457bab40-a577-4197-8a29-9711cc6cff5c\") " pod="opendatahub/maas-controller-6d844fcccb-h5ckv"
Jul 30 02:38:17.789079 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.788957    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcphb\" (UniqueName: \"kubernetes.io/projected/457bab40-a577-4197-8a29-9711cc6cff5c-kube-api-access-tcphb\") pod \"maas-controller-6d844fcccb-h5ckv\" (UID: \"457bab40-a577-4197-8a29-9711cc6cff5c\") " pod="opendatahub/maas-controller-6d844fcccb-h5ckv"
Jul 30 02:38:17.789079 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.788988    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptvgv\" (UniqueName: \"kubernetes.io/projected/620df212-4dc4-4298-9551-bbc5b71d84d2-kube-api-access-ptvgv\") pod \"maas-controller-6899b54cb8-c9l7w\" (UID: \"620df212-4dc4-4298-9551-bbc5b71d84d2\") " pod="opendatahub/maas-controller-6899b54cb8-c9l7w"
Jul 30 02:38:17.791418 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.791386    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/620df212-4dc4-4298-9551-bbc5b71d84d2-webhook-cert\") pod \"maas-controller-6899b54cb8-c9l7w\" (UID: \"620df212-4dc4-4298-9551-bbc5b71d84d2\") " pod="opendatahub/maas-controller-6899b54cb8-c9l7w"
Jul 30 02:38:17.791540 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.791444    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/457bab40-a577-4197-8a29-9711cc6cff5c-webhook-cert\") pod \"maas-controller-6d844fcccb-h5ckv\" (UID: \"457bab40-a577-4197-8a29-9711cc6cff5c\") " pod="opendatahub/maas-controller-6d844fcccb-h5ckv"
Jul 30 02:38:17.798448 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.798430    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptvgv\" (UniqueName: \"kubernetes.io/projected/620df212-4dc4-4298-9551-bbc5b71d84d2-kube-api-access-ptvgv\") pod \"maas-controller-6899b54cb8-c9l7w\" (UID: \"620df212-4dc4-4298-9551-bbc5b71d84d2\") " pod="opendatahub/maas-controller-6899b54cb8-c9l7w"
Jul 30 02:38:17.798903 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.798880    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcphb\" (UniqueName: \"kubernetes.io/projected/457bab40-a577-4197-8a29-9711cc6cff5c-kube-api-access-tcphb\") pod \"maas-controller-6d844fcccb-h5ckv\" (UID: \"457bab40-a577-4197-8a29-9711cc6cff5c\") " pod="opendatahub/maas-controller-6d844fcccb-h5ckv"
Jul 30 02:38:17.907647 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.907618    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d844fcccb-h5ckv"
Jul 30 02:38:17.995409 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:17.995382    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6899b54cb8-c9l7w"
Jul 30 02:38:18.063653 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:38:18.063624    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457bab40_a577_4197_8a29_9711cc6cff5c.slice/crio-61618bfef94185cd989b6b815c7bb64b339cc95bcbbc03ccb422480333e2b7ce WatchSource:0}: Error finding container 61618bfef94185cd989b6b815c7bb64b339cc95bcbbc03ccb422480333e2b7ce: Status 404 returned error can't find the container with id 61618bfef94185cd989b6b815c7bb64b339cc95bcbbc03ccb422480333e2b7ce
Jul 30 02:38:18.063653 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:18.063636    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d844fcccb-h5ckv"]
Jul 30 02:38:18.160678 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:18.160655    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6899b54cb8-c9l7w"]
Jul 30 02:38:18.162184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:38:18.162142    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620df212_4dc4_4298_9551_bbc5b71d84d2.slice/crio-494d2f930cd06a67ec12d14eaf90377c0492f5beeec9442891159eb85717fda3 WatchSource:0}: Error finding container 494d2f930cd06a67ec12d14eaf90377c0492f5beeec9442891159eb85717fda3: Status 404 returned error can't find the container with id 494d2f930cd06a67ec12d14eaf90377c0492f5beeec9442891159eb85717fda3
Jul 30 02:38:18.815018 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:18.814981    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6899b54cb8-c9l7w" event={"ID":"620df212-4dc4-4298-9551-bbc5b71d84d2","Type":"ContainerStarted","Data":"494d2f930cd06a67ec12d14eaf90377c0492f5beeec9442891159eb85717fda3"}
Jul 30 02:38:18.816187 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:18.816153    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d844fcccb-h5ckv" event={"ID":"457bab40-a577-4197-8a29-9711cc6cff5c","Type":"ContainerStarted","Data":"61618bfef94185cd989b6b815c7bb64b339cc95bcbbc03ccb422480333e2b7ce"}
Jul 30 02:38:20.829844 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:20.829807    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6899b54cb8-c9l7w" event={"ID":"620df212-4dc4-4298-9551-bbc5b71d84d2","Type":"ContainerStarted","Data":"bb75141a1b6da875f68b335cb7373e60a94f5684bf6d7af6ad5eb0c80e687444"}
Jul 30 02:38:20.830414 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:20.829870    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6899b54cb8-c9l7w"
Jul 30 02:38:20.831318 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:20.831296    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d844fcccb-h5ckv" event={"ID":"457bab40-a577-4197-8a29-9711cc6cff5c","Type":"ContainerStarted","Data":"cc330d37a1ff50d731e638f8f085be334d89b65b755ec4a3ffdb0251db63b326"}
Jul 30 02:38:20.831434 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:20.831344    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d844fcccb-h5ckv"
Jul 30 02:38:20.852552 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:20.852505    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6899b54cb8-c9l7w" podStartSLOduration=1.814278997 podStartE2EDuration="3.852488319s" totalImagesPullingTime="2.038209322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:38:20.85003556 +0000 UTC m=+736.496818966" watchObservedRunningTime="2026-07-30 02:38:20.852488319 +0000 UTC m=+736.499271760"
Jul 30 02:38:20.867101 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:20.867059    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d844fcccb-h5ckv" podStartSLOduration=1.724189005 podStartE2EDuration="3.867044506s" totalImagesPullingTime="2.142855501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:38:20.866283041 +0000 UTC m=+736.513066458" watchObservedRunningTime="2026-07-30 02:38:20.867044506 +0000 UTC m=+736.513827918"
Jul 30 02:38:31.838668 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:31.838637    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6899b54cb8-c9l7w"
Jul 30 02:38:31.839205 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:31.838706    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d844fcccb-h5ckv"
Jul 30 02:38:31.900561 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:31.900525    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d844fcccb-h5ckv"]
Jul 30 02:38:31.900758 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:31.900736    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6d844fcccb-h5ckv" podUID="457bab40-a577-4197-8a29-9711cc6cff5c" containerName="manager" containerID="cri-o://cc330d37a1ff50d731e638f8f085be334d89b65b755ec4a3ffdb0251db63b326" gracePeriod=30
Jul 30 02:38:32.155385 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:32.155361    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d844fcccb-h5ckv"
Jul 30 02:38:32.219452 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:32.219417    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcphb\" (UniqueName: \"kubernetes.io/projected/457bab40-a577-4197-8a29-9711cc6cff5c-kube-api-access-tcphb\") pod \"457bab40-a577-4197-8a29-9711cc6cff5c\" (UID: \"457bab40-a577-4197-8a29-9711cc6cff5c\") "
Jul 30 02:38:32.219615 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:32.219499    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/457bab40-a577-4197-8a29-9711cc6cff5c-webhook-cert\") pod \"457bab40-a577-4197-8a29-9711cc6cff5c\" (UID: \"457bab40-a577-4197-8a29-9711cc6cff5c\") "
Jul 30 02:38:32.221816 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:32.221786    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457bab40-a577-4197-8a29-9711cc6cff5c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "457bab40-a577-4197-8a29-9711cc6cff5c" (UID: "457bab40-a577-4197-8a29-9711cc6cff5c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 02:38:32.221816 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:32.221789    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457bab40-a577-4197-8a29-9711cc6cff5c-kube-api-access-tcphb" (OuterVolumeSpecName: "kube-api-access-tcphb") pod "457bab40-a577-4197-8a29-9711cc6cff5c" (UID: "457bab40-a577-4197-8a29-9711cc6cff5c"). InnerVolumeSpecName "kube-api-access-tcphb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:38:32.320540 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:32.320497    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcphb\" (UniqueName: \"kubernetes.io/projected/457bab40-a577-4197-8a29-9711cc6cff5c-kube-api-access-tcphb\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:38:32.320540 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:32.320540    2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/457bab40-a577-4197-8a29-9711cc6cff5c-webhook-cert\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:38:32.891845 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:32.891806    2565 generic.go:356] "Generic (PLEG): container finished" podID="457bab40-a577-4197-8a29-9711cc6cff5c" containerID="cc330d37a1ff50d731e638f8f085be334d89b65b755ec4a3ffdb0251db63b326" exitCode=0
Jul 30 02:38:32.892275 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:32.891852    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d844fcccb-h5ckv"
Jul 30 02:38:32.892275 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:32.891855    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d844fcccb-h5ckv" event={"ID":"457bab40-a577-4197-8a29-9711cc6cff5c","Type":"ContainerDied","Data":"cc330d37a1ff50d731e638f8f085be334d89b65b755ec4a3ffdb0251db63b326"}
Jul 30 02:38:32.892275 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:32.891907    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d844fcccb-h5ckv" event={"ID":"457bab40-a577-4197-8a29-9711cc6cff5c","Type":"ContainerDied","Data":"61618bfef94185cd989b6b815c7bb64b339cc95bcbbc03ccb422480333e2b7ce"}
Jul 30 02:38:32.892275 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:32.891937    2565 scope.go:117] "RemoveContainer" containerID="cc330d37a1ff50d731e638f8f085be334d89b65b755ec4a3ffdb0251db63b326"
Jul 30 02:38:32.900799 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:32.900782    2565 scope.go:117] "RemoveContainer" containerID="cc330d37a1ff50d731e638f8f085be334d89b65b755ec4a3ffdb0251db63b326"
Jul 30 02:38:32.901057 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:38:32.901037    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc330d37a1ff50d731e638f8f085be334d89b65b755ec4a3ffdb0251db63b326\": container with ID starting with cc330d37a1ff50d731e638f8f085be334d89b65b755ec4a3ffdb0251db63b326 not found: ID does not exist" containerID="cc330d37a1ff50d731e638f8f085be334d89b65b755ec4a3ffdb0251db63b326"
Jul 30 02:38:32.901108 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:32.901069    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc330d37a1ff50d731e638f8f085be334d89b65b755ec4a3ffdb0251db63b326"} err="failed to get container status \"cc330d37a1ff50d731e638f8f085be334d89b65b755ec4a3ffdb0251db63b326\": rpc error: code = NotFound desc = could not find container \"cc330d37a1ff50d731e638f8f085be334d89b65b755ec4a3ffdb0251db63b326\": container with ID starting with cc330d37a1ff50d731e638f8f085be334d89b65b755ec4a3ffdb0251db63b326 not found: ID does not exist"
Jul 30 02:38:32.909756 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:32.909729    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d844fcccb-h5ckv"]
Jul 30 02:38:32.912857 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:32.912835    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d844fcccb-h5ckv"]
Jul 30 02:38:34.800710 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:34.800675    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="457bab40-a577-4197-8a29-9711cc6cff5c" path="/var/lib/kubelet/pods/457bab40-a577-4197-8a29-9711cc6cff5c/volumes"
Jul 30 02:38:53.720403 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:53.720363    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86659d4467-rv8gb"]
Jul 30 02:38:53.721275 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:53.721253    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="457bab40-a577-4197-8a29-9711cc6cff5c" containerName="manager"
Jul 30 02:38:53.721275 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:53.721277    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="457bab40-a577-4197-8a29-9711cc6cff5c" containerName="manager"
Jul 30 02:38:53.721508 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:53.721494    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="457bab40-a577-4197-8a29-9711cc6cff5c" containerName="manager"
Jul 30 02:38:53.729766 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:53.729743    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86659d4467-rv8gb"
Jul 30 02:38:53.733072 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:53.733043    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v5sbq\""
Jul 30 02:38:53.733072 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:53.733045    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 30 02:38:53.734251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:53.734228    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86659d4467-rv8gb"]
Jul 30 02:38:53.921535 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:53.921499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68bb827-8a52-4b49-b3b3-55c6c355d258-maas-api-tls\") pod \"maas-api-86659d4467-rv8gb\" (UID: \"e68bb827-8a52-4b49-b3b3-55c6c355d258\") " pod="odh-ai-gateway-infra/maas-api-86659d4467-rv8gb"
Jul 30 02:38:53.921717 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:53.921598    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68bb827-8a52-4b49-b3b3-55c6c355d258-openshift-service-ca\") pod \"maas-api-86659d4467-rv8gb\" (UID: \"e68bb827-8a52-4b49-b3b3-55c6c355d258\") " pod="odh-ai-gateway-infra/maas-api-86659d4467-rv8gb"
Jul 30 02:38:53.921766 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:53.921726    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mksgm\" (UniqueName: \"kubernetes.io/projected/e68bb827-8a52-4b49-b3b3-55c6c355d258-kube-api-access-mksgm\") pod \"maas-api-86659d4467-rv8gb\" (UID: \"e68bb827-8a52-4b49-b3b3-55c6c355d258\") " pod="odh-ai-gateway-infra/maas-api-86659d4467-rv8gb"
Jul 30 02:38:54.023162 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.023079    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68bb827-8a52-4b49-b3b3-55c6c355d258-maas-api-tls\") pod \"maas-api-86659d4467-rv8gb\" (UID: \"e68bb827-8a52-4b49-b3b3-55c6c355d258\") " pod="odh-ai-gateway-infra/maas-api-86659d4467-rv8gb"
Jul 30 02:38:54.023162 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.023121    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68bb827-8a52-4b49-b3b3-55c6c355d258-openshift-service-ca\") pod \"maas-api-86659d4467-rv8gb\" (UID: \"e68bb827-8a52-4b49-b3b3-55c6c355d258\") " pod="odh-ai-gateway-infra/maas-api-86659d4467-rv8gb"
Jul 30 02:38:54.023376 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.023201    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mksgm\" (UniqueName: \"kubernetes.io/projected/e68bb827-8a52-4b49-b3b3-55c6c355d258-kube-api-access-mksgm\") pod \"maas-api-86659d4467-rv8gb\" (UID: \"e68bb827-8a52-4b49-b3b3-55c6c355d258\") " pod="odh-ai-gateway-infra/maas-api-86659d4467-rv8gb"
Jul 30 02:38:54.023893 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.023862    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68bb827-8a52-4b49-b3b3-55c6c355d258-openshift-service-ca\") pod \"maas-api-86659d4467-rv8gb\" (UID: \"e68bb827-8a52-4b49-b3b3-55c6c355d258\") " pod="odh-ai-gateway-infra/maas-api-86659d4467-rv8gb"
Jul 30 02:38:54.026028 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.026005    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68bb827-8a52-4b49-b3b3-55c6c355d258-maas-api-tls\") pod \"maas-api-86659d4467-rv8gb\" (UID: \"e68bb827-8a52-4b49-b3b3-55c6c355d258\") " pod="odh-ai-gateway-infra/maas-api-86659d4467-rv8gb"
Jul 30 02:38:54.034726 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.034704    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mksgm\" (UniqueName: \"kubernetes.io/projected/e68bb827-8a52-4b49-b3b3-55c6c355d258-kube-api-access-mksgm\") pod \"maas-api-86659d4467-rv8gb\" (UID: \"e68bb827-8a52-4b49-b3b3-55c6c355d258\") " pod="odh-ai-gateway-infra/maas-api-86659d4467-rv8gb"
Jul 30 02:38:54.040452 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.040429    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86659d4467-rv8gb"
Jul 30 02:38:54.191743 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.191710    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86659d4467-rv8gb"]
Jul 30 02:38:54.195150 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:38:54.195120    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68bb827_8a52_4b49_b3b3_55c6c355d258.slice/crio-39b947de500fbb3b96914e264c3707371fa0181a510e96591fbf4537406314bf WatchSource:0}: Error finding container 39b947de500fbb3b96914e264c3707371fa0181a510e96591fbf4537406314bf: Status 404 returned error can't find the container with id 39b947de500fbb3b96914e264c3707371fa0181a510e96591fbf4537406314bf
Jul 30 02:38:54.206854 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.206829    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-wg4rd"]
Jul 30 02:38:54.212883 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.212864    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-wg4rd"
Jul 30 02:38:54.215855 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.215833    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-f87p6\""
Jul 30 02:38:54.215855 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.215852    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 30 02:38:54.221699 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.221665    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-wg4rd"]
Jul 30 02:38:54.326097 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.326061    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gxdh\" (UniqueName: \"kubernetes.io/projected/0c013301-ce62-4de8-87b6-4e9e6ed43dba-kube-api-access-2gxdh\") pod \"payload-processing-5cc6cd444b-wg4rd\" (UID: \"0c013301-ce62-4de8-87b6-4e9e6ed43dba\") " pod="openshift-ingress/payload-processing-5cc6cd444b-wg4rd"
Jul 30 02:38:54.326307 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.326147    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c013301-ce62-4de8-87b6-4e9e6ed43dba-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-wg4rd\" (UID: \"0c013301-ce62-4de8-87b6-4e9e6ed43dba\") " pod="openshift-ingress/payload-processing-5cc6cd444b-wg4rd"
Jul 30 02:38:54.427472 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.427443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gxdh\" (UniqueName: \"kubernetes.io/projected/0c013301-ce62-4de8-87b6-4e9e6ed43dba-kube-api-access-2gxdh\") pod \"payload-processing-5cc6cd444b-wg4rd\" (UID: \"0c013301-ce62-4de8-87b6-4e9e6ed43dba\") " pod="openshift-ingress/payload-processing-5cc6cd444b-wg4rd"
Jul 30 02:38:54.427627 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.427526    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c013301-ce62-4de8-87b6-4e9e6ed43dba-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-wg4rd\" (UID: \"0c013301-ce62-4de8-87b6-4e9e6ed43dba\") " pod="openshift-ingress/payload-processing-5cc6cd444b-wg4rd"
Jul 30 02:38:54.428315 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.428293    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c013301-ce62-4de8-87b6-4e9e6ed43dba-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-wg4rd\" (UID: \"0c013301-ce62-4de8-87b6-4e9e6ed43dba\") " pod="openshift-ingress/payload-processing-5cc6cd444b-wg4rd"
Jul 30 02:38:54.435731 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.435697    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gxdh\" (UniqueName: \"kubernetes.io/projected/0c013301-ce62-4de8-87b6-4e9e6ed43dba-kube-api-access-2gxdh\") pod \"payload-processing-5cc6cd444b-wg4rd\" (UID: \"0c013301-ce62-4de8-87b6-4e9e6ed43dba\") " pod="openshift-ingress/payload-processing-5cc6cd444b-wg4rd"
Jul 30 02:38:54.524974 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.524939    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-wg4rd"
Jul 30 02:38:54.659809 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:54.659786    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-wg4rd"]
Jul 30 02:38:54.661467 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:38:54.661439    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c013301_ce62_4de8_87b6_4e9e6ed43dba.slice/crio-52982ae2c2d71294e1a1b281800d503b8f4d77f48193c38ee0b6cbe052234e13 WatchSource:0}: Error finding container 52982ae2c2d71294e1a1b281800d503b8f4d77f48193c38ee0b6cbe052234e13: Status 404 returned error can't find the container with id 52982ae2c2d71294e1a1b281800d503b8f4d77f48193c38ee0b6cbe052234e13
Jul 30 02:38:55.018515 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:55.018375    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86659d4467-rv8gb" event={"ID":"e68bb827-8a52-4b49-b3b3-55c6c355d258","Type":"ContainerStarted","Data":"39b947de500fbb3b96914e264c3707371fa0181a510e96591fbf4537406314bf"}
Jul 30 02:38:55.020993 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:55.020924    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-wg4rd" event={"ID":"0c013301-ce62-4de8-87b6-4e9e6ed43dba","Type":"ContainerStarted","Data":"52982ae2c2d71294e1a1b281800d503b8f4d77f48193c38ee0b6cbe052234e13"}
Jul 30 02:38:59.047463 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:59.047408    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86659d4467-rv8gb" event={"ID":"e68bb827-8a52-4b49-b3b3-55c6c355d258","Type":"ContainerStarted","Data":"5b08b552e73b40b1a7b3bc9cb53adcc0f892f1955d9059784b459e301acc4fd1"}
Jul 30 02:38:59.047788 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:59.047572    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86659d4467-rv8gb"
Jul 30 02:38:59.048886 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:59.048865    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-wg4rd" event={"ID":"0c013301-ce62-4de8-87b6-4e9e6ed43dba","Type":"ContainerStarted","Data":"1450f54ccb66bb4dae3d62e5eb880db65fd60e1f2b0b4ae3c290c7c05625e329"}
Jul 30 02:38:59.048956 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:59.048920    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-wg4rd"
Jul 30 02:38:59.065323 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:59.065218    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86659d4467-rv8gb" podStartSLOduration=2.299167075 podStartE2EDuration="6.065203252s" totalImagesPullingTime="3.766036177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:38:59.064666089 +0000 UTC m=+774.711449515" watchObservedRunningTime="2026-07-30 02:38:59.065203252 +0000 UTC m=+774.711986659"
Jul 30 02:38:59.084341 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:38:59.084292    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-wg4rd" podStartSLOduration=1.783061708 podStartE2EDuration="5.084278849s" totalImagesPullingTime="3.301217141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:38:59.082325239 +0000 UTC m=+774.729108657" watchObservedRunningTime="2026-07-30 02:38:59.084278849 +0000 UTC m=+774.731062253"
Jul 30 02:39:02.067122 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:02.067029    2565 generic.go:356] "Generic (PLEG): container finished" podID="cd921d07-bc19-45b2-b724-43dce72b1d84" containerID="9f928dacae49f09ba1a19eb19030570cfb1a8b41556a2d9d9088f7e1946a9fbc" exitCode=0
Jul 30 02:39:02.067122 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:02.067100    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j" event={"ID":"cd921d07-bc19-45b2-b724-43dce72b1d84","Type":"ContainerDied","Data":"9f928dacae49f09ba1a19eb19030570cfb1a8b41556a2d9d9088f7e1946a9fbc"}
Jul 30 02:39:02.067577 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:02.067498    2565 scope.go:117] "RemoveContainer" containerID="9f928dacae49f09ba1a19eb19030570cfb1a8b41556a2d9d9088f7e1946a9fbc"
Jul 30 02:39:02.068486 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:02.068459    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:39:02.068553 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:02.068526    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:39:02.068590 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:02.068560    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:39:03.075508 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:03.075464    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j" event={"ID":"cd921d07-bc19-45b2-b724-43dce72b1d84","Type":"ContainerStarted","Data":"ab0968fe7ef72ec616f5326b2a52211c1374b89ec331f4dbd699195e0097929b"}
Jul 30 02:39:04.074847 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:04.074805    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:39:04.075207 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:04.075145    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9vf5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Jul 30 02:39:04.075336 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:04.075248    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j" podUID="cd921d07-bc19-45b2-b724-43dce72b1d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Jul 30 02:39:05.060948 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:05.060922    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86659d4467-rv8gb"
Jul 30 02:39:05.074389 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:05.074352    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9vf5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Jul 30 02:39:05.074563 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:05.074419    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j" podUID="cd921d07-bc19-45b2-b724-43dce72b1d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Jul 30 02:39:06.074438 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:06.074398    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9vf5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Jul 30 02:39:06.074932 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:06.074467    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j" podUID="cd921d07-bc19-45b2-b724-43dce72b1d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Jul 30 02:39:07.074442 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:07.074399    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9vf5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Jul 30 02:39:07.075016 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:07.074985    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j" podUID="cd921d07-bc19-45b2-b724-43dce72b1d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Jul 30 02:39:08.078993 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:08.078958    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:39:08.079484 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:08.079377    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:39:08.079986 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:08.079965    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9vf5j"
Jul 30 02:39:10.057064 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:10.057038    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-wg4rd"
Jul 30 02:39:46.040888 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.040854    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"]
Jul 30 02:39:46.044567 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.044549    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:46.050777 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.050749    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 30 02:39:46.050935 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.050890    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2mqsd\""
Jul 30 02:39:46.050935 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.050755    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 30 02:39:46.051221 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.051204    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 30 02:39:46.054251 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.054230    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"]
Jul 30 02:39:46.181080 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.181046    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc883c92-489d-422d-8ef2-5ba476ab0291-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bts5f\" (UID: \"bc883c92-489d-422d-8ef2-5ba476ab0291\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:46.181080 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.181081    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc883c92-489d-422d-8ef2-5ba476ab0291-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bts5f\" (UID: \"bc883c92-489d-422d-8ef2-5ba476ab0291\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:46.181356 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.181181    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpd5g\" (UniqueName: \"kubernetes.io/projected/bc883c92-489d-422d-8ef2-5ba476ab0291-kube-api-access-mpd5g\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bts5f\" (UID: \"bc883c92-489d-422d-8ef2-5ba476ab0291\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:46.181356 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.181245    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc883c92-489d-422d-8ef2-5ba476ab0291-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bts5f\" (UID: \"bc883c92-489d-422d-8ef2-5ba476ab0291\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:46.181356 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.181263    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc883c92-489d-422d-8ef2-5ba476ab0291-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bts5f\" (UID: \"bc883c92-489d-422d-8ef2-5ba476ab0291\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:46.181356 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.181326    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc883c92-489d-422d-8ef2-5ba476ab0291-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bts5f\" (UID: \"bc883c92-489d-422d-8ef2-5ba476ab0291\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:46.282628 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.282591    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpd5g\" (UniqueName: \"kubernetes.io/projected/bc883c92-489d-422d-8ef2-5ba476ab0291-kube-api-access-mpd5g\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bts5f\" (UID: \"bc883c92-489d-422d-8ef2-5ba476ab0291\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:46.282819 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.282654    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc883c92-489d-422d-8ef2-5ba476ab0291-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bts5f\" (UID: \"bc883c92-489d-422d-8ef2-5ba476ab0291\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:46.282819 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.282672    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc883c92-489d-422d-8ef2-5ba476ab0291-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bts5f\" (UID: \"bc883c92-489d-422d-8ef2-5ba476ab0291\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:46.282819 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.282697    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc883c92-489d-422d-8ef2-5ba476ab0291-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bts5f\" (UID: \"bc883c92-489d-422d-8ef2-5ba476ab0291\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:46.282819 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.282732    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc883c92-489d-422d-8ef2-5ba476ab0291-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bts5f\" (UID: \"bc883c92-489d-422d-8ef2-5ba476ab0291\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:46.283035 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.282870    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc883c92-489d-422d-8ef2-5ba476ab0291-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bts5f\" (UID: \"bc883c92-489d-422d-8ef2-5ba476ab0291\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:46.283096 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.283062    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc883c92-489d-422d-8ef2-5ba476ab0291-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bts5f\" (UID: \"bc883c92-489d-422d-8ef2-5ba476ab0291\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:46.283231 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.283196    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc883c92-489d-422d-8ef2-5ba476ab0291-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bts5f\" (UID: \"bc883c92-489d-422d-8ef2-5ba476ab0291\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:46.283362 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.283342    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc883c92-489d-422d-8ef2-5ba476ab0291-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bts5f\" (UID: \"bc883c92-489d-422d-8ef2-5ba476ab0291\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:46.285263 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.285240    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc883c92-489d-422d-8ef2-5ba476ab0291-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bts5f\" (UID: \"bc883c92-489d-422d-8ef2-5ba476ab0291\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:46.285356 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.285343    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc883c92-489d-422d-8ef2-5ba476ab0291-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bts5f\" (UID: \"bc883c92-489d-422d-8ef2-5ba476ab0291\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:46.291201 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.291127    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpd5g\" (UniqueName: \"kubernetes.io/projected/bc883c92-489d-422d-8ef2-5ba476ab0291-kube-api-access-mpd5g\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bts5f\" (UID: \"bc883c92-489d-422d-8ef2-5ba476ab0291\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:46.355435 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.355403    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:46.489135 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:46.489109    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"]
Jul 30 02:39:46.491938 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:39:46.491908    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc883c92_489d_422d_8ef2_5ba476ab0291.slice/crio-ffd0a5fac2b0f5630c56c9259858f631f83be10b0839bc69ef9ef6d7f4e557c5 WatchSource:0}: Error finding container ffd0a5fac2b0f5630c56c9259858f631f83be10b0839bc69ef9ef6d7f4e557c5: Status 404 returned error can't find the container with id ffd0a5fac2b0f5630c56c9259858f631f83be10b0839bc69ef9ef6d7f4e557c5
Jul 30 02:39:47.324033 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:47.323995    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f" event={"ID":"bc883c92-489d-422d-8ef2-5ba476ab0291","Type":"ContainerStarted","Data":"ffd0a5fac2b0f5630c56c9259858f631f83be10b0839bc69ef9ef6d7f4e557c5"}
Jul 30 02:39:48.331256 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:48.331221    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f" event={"ID":"bc883c92-489d-422d-8ef2-5ba476ab0291","Type":"ContainerStarted","Data":"cefd6970b396343a9030a01f948cdbb0c35ef4db4b60b2f70b1d2ffba4a8da37"}
Jul 30 02:39:48.351292 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:48.351245    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f" podStartSLOduration=1.247080854 podStartE2EDuration="2.351230236s" totalImagesPullingTime="1.104149382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:39:48.349579724 +0000 UTC m=+823.996363128" watchObservedRunningTime="2026-07-30 02:39:48.351230236 +0000 UTC m=+823.998013640"
Jul 30 02:39:56.356535 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:56.356493    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:56.357018 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:56.356559    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:56.369542 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:56.369518    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:39:56.387638 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:39:56.387606    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bts5f"
Jul 30 02:41:40.451698 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:41:40.451663    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5bd6b597bd-ww9hj"]
Jul 30 02:41:40.452367 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:41:40.451865    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5bd6b597bd-ww9hj" podUID="d637db25-c915-479d-a504-6e1234ef4783" containerName="authorino" containerID="cri-o://3709313825a600e2ce57e841d51942756b13b6cb5daf28a45ad3f7e1962257d6" gracePeriod=30
Jul 30 02:41:40.697477 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:41:40.697453    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bd6b597bd-ww9hj"
Jul 30 02:41:40.833132 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:41:40.833100    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szbgh\" (UniqueName: \"kubernetes.io/projected/d637db25-c915-479d-a504-6e1234ef4783-kube-api-access-szbgh\") pod \"d637db25-c915-479d-a504-6e1234ef4783\" (UID: \"d637db25-c915-479d-a504-6e1234ef4783\") "
Jul 30 02:41:40.833314 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:41:40.833181    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d637db25-c915-479d-a504-6e1234ef4783-tls-cert\") pod \"d637db25-c915-479d-a504-6e1234ef4783\" (UID: \"d637db25-c915-479d-a504-6e1234ef4783\") "
Jul 30 02:41:40.835444 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:41:40.835420    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d637db25-c915-479d-a504-6e1234ef4783-kube-api-access-szbgh" (OuterVolumeSpecName: "kube-api-access-szbgh") pod "d637db25-c915-479d-a504-6e1234ef4783" (UID: "d637db25-c915-479d-a504-6e1234ef4783"). InnerVolumeSpecName "kube-api-access-szbgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:41:40.843486 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:41:40.843460    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d637db25-c915-479d-a504-6e1234ef4783-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d637db25-c915-479d-a504-6e1234ef4783" (UID: "d637db25-c915-479d-a504-6e1234ef4783"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 02:41:40.934605 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:41:40.934576    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szbgh\" (UniqueName: \"kubernetes.io/projected/d637db25-c915-479d-a504-6e1234ef4783-kube-api-access-szbgh\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:41:40.934605 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:41:40.934602    2565 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d637db25-c915-479d-a504-6e1234ef4783-tls-cert\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:41:40.955824 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:41:40.955791    2565 generic.go:356] "Generic (PLEG): container finished" podID="d637db25-c915-479d-a504-6e1234ef4783" containerID="3709313825a600e2ce57e841d51942756b13b6cb5daf28a45ad3f7e1962257d6" exitCode=0
Jul 30 02:41:40.955949 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:41:40.955849    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bd6b597bd-ww9hj"
Jul 30 02:41:40.955949 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:41:40.955884    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bd6b597bd-ww9hj" event={"ID":"d637db25-c915-479d-a504-6e1234ef4783","Type":"ContainerDied","Data":"3709313825a600e2ce57e841d51942756b13b6cb5daf28a45ad3f7e1962257d6"}
Jul 30 02:41:40.955949 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:41:40.955934    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bd6b597bd-ww9hj" event={"ID":"d637db25-c915-479d-a504-6e1234ef4783","Type":"ContainerDied","Data":"79f11794111bada36564d1e934bd057a9deaa06263c7d232613e84f8db78bd01"}
Jul 30 02:41:40.956089 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:41:40.955955    2565 scope.go:117] "RemoveContainer" containerID="3709313825a600e2ce57e841d51942756b13b6cb5daf28a45ad3f7e1962257d6"
Jul 30 02:41:40.966158 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:41:40.966137    2565 scope.go:117] "RemoveContainer" containerID="3709313825a600e2ce57e841d51942756b13b6cb5daf28a45ad3f7e1962257d6"
Jul 30 02:41:40.966500 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:41:40.966478    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3709313825a600e2ce57e841d51942756b13b6cb5daf28a45ad3f7e1962257d6\": container with ID starting with 3709313825a600e2ce57e841d51942756b13b6cb5daf28a45ad3f7e1962257d6 not found: ID does not exist" containerID="3709313825a600e2ce57e841d51942756b13b6cb5daf28a45ad3f7e1962257d6"
Jul 30 02:41:40.966576 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:41:40.966509    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3709313825a600e2ce57e841d51942756b13b6cb5daf28a45ad3f7e1962257d6"} err="failed to get container status \"3709313825a600e2ce57e841d51942756b13b6cb5daf28a45ad3f7e1962257d6\": rpc error: code = NotFound desc = could not find container \"3709313825a600e2ce57e841d51942756b13b6cb5daf28a45ad3f7e1962257d6\": container with ID starting with 3709313825a600e2ce57e841d51942756b13b6cb5daf28a45ad3f7e1962257d6 not found: ID does not exist"
Jul 30 02:41:40.995513 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:41:40.995483    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5bd6b597bd-ww9hj"]
Jul 30 02:41:40.998357 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:41:40.998328    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5bd6b597bd-ww9hj"]
Jul 30 02:41:42.800722 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:41:42.800692    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d637db25-c915-479d-a504-6e1234ef4783" path="/var/lib/kubelet/pods/d637db25-c915-479d-a504-6e1234ef4783/volumes"
Jul 30 02:42:28.736422 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:42:28.736338    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6899b54cb8-c9l7w"]
Jul 30 02:42:28.737066 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:42:28.736549    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6899b54cb8-c9l7w" podUID="620df212-4dc4-4298-9551-bbc5b71d84d2" containerName="manager" containerID="cri-o://bb75141a1b6da875f68b335cb7373e60a94f5684bf6d7af6ad5eb0c80e687444" gracePeriod=30
Jul 30 02:42:28.987686 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:42:28.987626    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6899b54cb8-c9l7w"
Jul 30 02:42:29.072902 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:42:29.072870    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptvgv\" (UniqueName: \"kubernetes.io/projected/620df212-4dc4-4298-9551-bbc5b71d84d2-kube-api-access-ptvgv\") pod \"620df212-4dc4-4298-9551-bbc5b71d84d2\" (UID: \"620df212-4dc4-4298-9551-bbc5b71d84d2\") "
Jul 30 02:42:29.073094 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:42:29.073026    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/620df212-4dc4-4298-9551-bbc5b71d84d2-webhook-cert\") pod \"620df212-4dc4-4298-9551-bbc5b71d84d2\" (UID: \"620df212-4dc4-4298-9551-bbc5b71d84d2\") "
Jul 30 02:42:29.075099 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:42:29.075063    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/620df212-4dc4-4298-9551-bbc5b71d84d2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "620df212-4dc4-4298-9551-bbc5b71d84d2" (UID: "620df212-4dc4-4298-9551-bbc5b71d84d2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 02:42:29.075233 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:42:29.075150    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620df212-4dc4-4298-9551-bbc5b71d84d2-kube-api-access-ptvgv" (OuterVolumeSpecName: "kube-api-access-ptvgv") pod "620df212-4dc4-4298-9551-bbc5b71d84d2" (UID: "620df212-4dc4-4298-9551-bbc5b71d84d2"). InnerVolumeSpecName "kube-api-access-ptvgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:42:29.174022 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:42:29.173989    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptvgv\" (UniqueName: \"kubernetes.io/projected/620df212-4dc4-4298-9551-bbc5b71d84d2-kube-api-access-ptvgv\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:42:29.174022 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:42:29.174018    2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/620df212-4dc4-4298-9551-bbc5b71d84d2-webhook-cert\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:42:29.227113 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:42:29.227073    2565 generic.go:356] "Generic (PLEG): container finished" podID="620df212-4dc4-4298-9551-bbc5b71d84d2" containerID="bb75141a1b6da875f68b335cb7373e60a94f5684bf6d7af6ad5eb0c80e687444" exitCode=0
Jul 30 02:42:29.227300 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:42:29.227134    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6899b54cb8-c9l7w"
Jul 30 02:42:29.227300 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:42:29.227159    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6899b54cb8-c9l7w" event={"ID":"620df212-4dc4-4298-9551-bbc5b71d84d2","Type":"ContainerDied","Data":"bb75141a1b6da875f68b335cb7373e60a94f5684bf6d7af6ad5eb0c80e687444"}
Jul 30 02:42:29.227300 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:42:29.227257    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6899b54cb8-c9l7w" event={"ID":"620df212-4dc4-4298-9551-bbc5b71d84d2","Type":"ContainerDied","Data":"494d2f930cd06a67ec12d14eaf90377c0492f5beeec9442891159eb85717fda3"}
Jul 30 02:42:29.227300 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:42:29.227281    2565 scope.go:117] "RemoveContainer" containerID="bb75141a1b6da875f68b335cb7373e60a94f5684bf6d7af6ad5eb0c80e687444"
Jul 30 02:42:29.237005 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:42:29.236987    2565 scope.go:117] "RemoveContainer" containerID="bb75141a1b6da875f68b335cb7373e60a94f5684bf6d7af6ad5eb0c80e687444"
Jul 30 02:42:29.237318 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:42:29.237282    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb75141a1b6da875f68b335cb7373e60a94f5684bf6d7af6ad5eb0c80e687444\": container with ID starting with bb75141a1b6da875f68b335cb7373e60a94f5684bf6d7af6ad5eb0c80e687444 not found: ID does not exist" containerID="bb75141a1b6da875f68b335cb7373e60a94f5684bf6d7af6ad5eb0c80e687444"
Jul 30 02:42:29.237374 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:42:29.237337    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb75141a1b6da875f68b335cb7373e60a94f5684bf6d7af6ad5eb0c80e687444"} err="failed to get container status \"bb75141a1b6da875f68b335cb7373e60a94f5684bf6d7af6ad5eb0c80e687444\": rpc error: code = NotFound desc = could not find container \"bb75141a1b6da875f68b335cb7373e60a94f5684bf6d7af6ad5eb0c80e687444\": container with ID starting with bb75141a1b6da875f68b335cb7373e60a94f5684bf6d7af6ad5eb0c80e687444 not found: ID does not exist"
Jul 30 02:42:29.249893 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:42:29.249840    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6899b54cb8-c9l7w"]
Jul 30 02:42:29.254860 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:42:29.254836    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6899b54cb8-c9l7w"]
Jul 30 02:42:30.800572 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:42:30.800534    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="620df212-4dc4-4298-9551-bbc5b71d84d2" path="/var/lib/kubelet/pods/620df212-4dc4-4298-9551-bbc5b71d84d2/volumes"
Jul 30 02:43:30.053710 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.053677    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6899b54cb8-rxqjq"]
Jul 30 02:43:30.054189 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.054061    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d637db25-c915-479d-a504-6e1234ef4783" containerName="authorino"
Jul 30 02:43:30.054189 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.054072    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d637db25-c915-479d-a504-6e1234ef4783" containerName="authorino"
Jul 30 02:43:30.054189 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.054080    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="620df212-4dc4-4298-9551-bbc5b71d84d2" containerName="manager"
Jul 30 02:43:30.054189 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.054085    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="620df212-4dc4-4298-9551-bbc5b71d84d2" containerName="manager"
Jul 30 02:43:30.054189 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.054140    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d637db25-c915-479d-a504-6e1234ef4783" containerName="authorino"
Jul 30 02:43:30.054189 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.054149    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="620df212-4dc4-4298-9551-bbc5b71d84d2" containerName="manager"
Jul 30 02:43:30.057283 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.057262    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6899b54cb8-rxqjq"
Jul 30 02:43:30.061400 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.061362    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wxw25\""
Jul 30 02:43:30.061400 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.061390    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 30 02:43:30.061400 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.061399    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 30 02:43:30.067656 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.067629    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6899b54cb8-rxqjq"]
Jul 30 02:43:30.214234 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.214195    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c77f4d6c-0fce-4885-8a2f-00c5d946ffa8-webhook-cert\") pod \"maas-controller-6899b54cb8-rxqjq\" (UID: \"c77f4d6c-0fce-4885-8a2f-00c5d946ffa8\") " pod="opendatahub/maas-controller-6899b54cb8-rxqjq"
Jul 30 02:43:30.214234 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.214241    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smwpq\" (UniqueName: \"kubernetes.io/projected/c77f4d6c-0fce-4885-8a2f-00c5d946ffa8-kube-api-access-smwpq\") pod \"maas-controller-6899b54cb8-rxqjq\" (UID: \"c77f4d6c-0fce-4885-8a2f-00c5d946ffa8\") " pod="opendatahub/maas-controller-6899b54cb8-rxqjq"
Jul 30 02:43:30.315470 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.315376    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c77f4d6c-0fce-4885-8a2f-00c5d946ffa8-webhook-cert\") pod \"maas-controller-6899b54cb8-rxqjq\" (UID: \"c77f4d6c-0fce-4885-8a2f-00c5d946ffa8\") " pod="opendatahub/maas-controller-6899b54cb8-rxqjq"
Jul 30 02:43:30.315470 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.315422    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smwpq\" (UniqueName: \"kubernetes.io/projected/c77f4d6c-0fce-4885-8a2f-00c5d946ffa8-kube-api-access-smwpq\") pod \"maas-controller-6899b54cb8-rxqjq\" (UID: \"c77f4d6c-0fce-4885-8a2f-00c5d946ffa8\") " pod="opendatahub/maas-controller-6899b54cb8-rxqjq"
Jul 30 02:43:30.317908 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.317884    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c77f4d6c-0fce-4885-8a2f-00c5d946ffa8-webhook-cert\") pod \"maas-controller-6899b54cb8-rxqjq\" (UID: \"c77f4d6c-0fce-4885-8a2f-00c5d946ffa8\") " pod="opendatahub/maas-controller-6899b54cb8-rxqjq"
Jul 30 02:43:30.323622 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.323597    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smwpq\" (UniqueName: \"kubernetes.io/projected/c77f4d6c-0fce-4885-8a2f-00c5d946ffa8-kube-api-access-smwpq\") pod \"maas-controller-6899b54cb8-rxqjq\" (UID: \"c77f4d6c-0fce-4885-8a2f-00c5d946ffa8\") " pod="opendatahub/maas-controller-6899b54cb8-rxqjq"
Jul 30 02:43:30.370576 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.370549    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6899b54cb8-rxqjq"
Jul 30 02:43:30.506426 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.506396    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6899b54cb8-rxqjq"]
Jul 30 02:43:30.507944 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:43:30.507912    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77f4d6c_0fce_4885_8a2f_00c5d946ffa8.slice/crio-2ca7a16b354824c1d5fd242a536f34e0e4ee0a705a8c468294e664c002eab827 WatchSource:0}: Error finding container 2ca7a16b354824c1d5fd242a536f34e0e4ee0a705a8c468294e664c002eab827: Status 404 returned error can't find the container with id 2ca7a16b354824c1d5fd242a536f34e0e4ee0a705a8c468294e664c002eab827
Jul 30 02:43:30.509244 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.509221    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 02:43:30.568056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:30.567977    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6899b54cb8-rxqjq" event={"ID":"c77f4d6c-0fce-4885-8a2f-00c5d946ffa8","Type":"ContainerStarted","Data":"2ca7a16b354824c1d5fd242a536f34e0e4ee0a705a8c468294e664c002eab827"}
Jul 30 02:43:31.575153 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:31.575113    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6899b54cb8-rxqjq" event={"ID":"c77f4d6c-0fce-4885-8a2f-00c5d946ffa8","Type":"ContainerStarted","Data":"292b14efb38e082ceb727b1a8a9c6efd2694ec48dabc26beb818d75b2d3298c3"}
Jul 30 02:43:31.575591 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:31.575186    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6899b54cb8-rxqjq"
Jul 30 02:43:31.594029 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:31.593980    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6899b54cb8-rxqjq" podStartSLOduration=1.061453617 podStartE2EDuration="1.593961466s" totalImagesPullingTime="532.507849ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:43:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:43:31.591923811 +0000 UTC m=+1047.238707215" watchObservedRunningTime="2026-07-30 02:43:31.593961466 +0000 UTC m=+1047.240744870"
Jul 30 02:43:42.583405 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:42.583375    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6899b54cb8-rxqjq"
Jul 30 02:43:47.228118 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:47.228077    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6899b54cb8-rxqjq"]
Jul 30 02:43:47.228605 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:47.228312    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6899b54cb8-rxqjq" podUID="c77f4d6c-0fce-4885-8a2f-00c5d946ffa8" containerName="manager" containerID="cri-o://292b14efb38e082ceb727b1a8a9c6efd2694ec48dabc26beb818d75b2d3298c3" gracePeriod=30
Jul 30 02:43:47.477246 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:47.477220    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6899b54cb8-rxqjq"
Jul 30 02:43:47.562983 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:47.562951    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c77f4d6c-0fce-4885-8a2f-00c5d946ffa8-webhook-cert\") pod \"c77f4d6c-0fce-4885-8a2f-00c5d946ffa8\" (UID: \"c77f4d6c-0fce-4885-8a2f-00c5d946ffa8\") "
Jul 30 02:43:47.563201 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:47.563028    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smwpq\" (UniqueName: \"kubernetes.io/projected/c77f4d6c-0fce-4885-8a2f-00c5d946ffa8-kube-api-access-smwpq\") pod \"c77f4d6c-0fce-4885-8a2f-00c5d946ffa8\" (UID: \"c77f4d6c-0fce-4885-8a2f-00c5d946ffa8\") "
Jul 30 02:43:47.565233 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:47.565202    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c77f4d6c-0fce-4885-8a2f-00c5d946ffa8-kube-api-access-smwpq" (OuterVolumeSpecName: "kube-api-access-smwpq") pod "c77f4d6c-0fce-4885-8a2f-00c5d946ffa8" (UID: "c77f4d6c-0fce-4885-8a2f-00c5d946ffa8"). InnerVolumeSpecName "kube-api-access-smwpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:43:47.565335 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:47.565284    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c77f4d6c-0fce-4885-8a2f-00c5d946ffa8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c77f4d6c-0fce-4885-8a2f-00c5d946ffa8" (UID: "c77f4d6c-0fce-4885-8a2f-00c5d946ffa8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 02:43:47.661878 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:47.661846    2565 generic.go:356] "Generic (PLEG): container finished" podID="c77f4d6c-0fce-4885-8a2f-00c5d946ffa8" containerID="292b14efb38e082ceb727b1a8a9c6efd2694ec48dabc26beb818d75b2d3298c3" exitCode=0
Jul 30 02:43:47.662048 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:47.661925    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6899b54cb8-rxqjq"
Jul 30 02:43:47.662048 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:47.661928    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6899b54cb8-rxqjq" event={"ID":"c77f4d6c-0fce-4885-8a2f-00c5d946ffa8","Type":"ContainerDied","Data":"292b14efb38e082ceb727b1a8a9c6efd2694ec48dabc26beb818d75b2d3298c3"}
Jul 30 02:43:47.662048 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:47.662008    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6899b54cb8-rxqjq" event={"ID":"c77f4d6c-0fce-4885-8a2f-00c5d946ffa8","Type":"ContainerDied","Data":"2ca7a16b354824c1d5fd242a536f34e0e4ee0a705a8c468294e664c002eab827"}
Jul 30 02:43:47.662048 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:47.662031    2565 scope.go:117] "RemoveContainer" containerID="292b14efb38e082ceb727b1a8a9c6efd2694ec48dabc26beb818d75b2d3298c3"
Jul 30 02:43:47.663963 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:47.663945    2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c77f4d6c-0fce-4885-8a2f-00c5d946ffa8-webhook-cert\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:43:47.663963 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:47.663964    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smwpq\" (UniqueName: \"kubernetes.io/projected/c77f4d6c-0fce-4885-8a2f-00c5d946ffa8-kube-api-access-smwpq\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:43:47.671061 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:47.671046    2565 scope.go:117] "RemoveContainer" containerID="292b14efb38e082ceb727b1a8a9c6efd2694ec48dabc26beb818d75b2d3298c3"
Jul 30 02:43:47.671328 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:43:47.671303    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"292b14efb38e082ceb727b1a8a9c6efd2694ec48dabc26beb818d75b2d3298c3\": container with ID starting with 292b14efb38e082ceb727b1a8a9c6efd2694ec48dabc26beb818d75b2d3298c3 not found: ID does not exist" containerID="292b14efb38e082ceb727b1a8a9c6efd2694ec48dabc26beb818d75b2d3298c3"
Jul 30 02:43:47.671418 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:47.671340    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292b14efb38e082ceb727b1a8a9c6efd2694ec48dabc26beb818d75b2d3298c3"} err="failed to get container status \"292b14efb38e082ceb727b1a8a9c6efd2694ec48dabc26beb818d75b2d3298c3\": rpc error: code = NotFound desc = could not find container \"292b14efb38e082ceb727b1a8a9c6efd2694ec48dabc26beb818d75b2d3298c3\": container with ID starting with 292b14efb38e082ceb727b1a8a9c6efd2694ec48dabc26beb818d75b2d3298c3 not found: ID does not exist"
Jul 30 02:43:47.685709 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:47.685686    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6899b54cb8-rxqjq"]
Jul 30 02:43:47.688522 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:47.688498    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6899b54cb8-rxqjq"]
Jul 30 02:43:48.467021 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:48.466992    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6899b54cb8-4k7tc"]
Jul 30 02:43:48.467487 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:48.467382    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c77f4d6c-0fce-4885-8a2f-00c5d946ffa8" containerName="manager"
Jul 30 02:43:48.467487 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:48.467394    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c77f4d6c-0fce-4885-8a2f-00c5d946ffa8" containerName="manager"
Jul 30 02:43:48.467487 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:48.467451    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c77f4d6c-0fce-4885-8a2f-00c5d946ffa8" containerName="manager"
Jul 30 02:43:48.471688 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:48.471666    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6899b54cb8-4k7tc"
Jul 30 02:43:48.476079 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:48.476057    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 30 02:43:48.476229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:48.476082    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 30 02:43:48.476489 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:48.476468    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wxw25\""
Jul 30 02:43:48.479669 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:48.479613    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6899b54cb8-4k7tc"]
Jul 30 02:43:48.573224 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:48.573191    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4b1585e2-ab06-4fc4-ad31-c3cf9912ea2b-webhook-cert\") pod \"maas-controller-6899b54cb8-4k7tc\" (UID: \"4b1585e2-ab06-4fc4-ad31-c3cf9912ea2b\") " pod="opendatahub/maas-controller-6899b54cb8-4k7tc"
Jul 30 02:43:48.573382 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:48.573308    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4r4p\" (UniqueName: \"kubernetes.io/projected/4b1585e2-ab06-4fc4-ad31-c3cf9912ea2b-kube-api-access-f4r4p\") pod \"maas-controller-6899b54cb8-4k7tc\" (UID: \"4b1585e2-ab06-4fc4-ad31-c3cf9912ea2b\") " pod="opendatahub/maas-controller-6899b54cb8-4k7tc"
Jul 30 02:43:48.674051 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:48.674014    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4r4p\" (UniqueName: \"kubernetes.io/projected/4b1585e2-ab06-4fc4-ad31-c3cf9912ea2b-kube-api-access-f4r4p\") pod \"maas-controller-6899b54cb8-4k7tc\" (UID: \"4b1585e2-ab06-4fc4-ad31-c3cf9912ea2b\") " pod="opendatahub/maas-controller-6899b54cb8-4k7tc"
Jul 30 02:43:48.674246 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:48.674094    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4b1585e2-ab06-4fc4-ad31-c3cf9912ea2b-webhook-cert\") pod \"maas-controller-6899b54cb8-4k7tc\" (UID: \"4b1585e2-ab06-4fc4-ad31-c3cf9912ea2b\") " pod="opendatahub/maas-controller-6899b54cb8-4k7tc"
Jul 30 02:43:48.676729 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:48.676706    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4b1585e2-ab06-4fc4-ad31-c3cf9912ea2b-webhook-cert\") pod \"maas-controller-6899b54cb8-4k7tc\" (UID: \"4b1585e2-ab06-4fc4-ad31-c3cf9912ea2b\") " pod="opendatahub/maas-controller-6899b54cb8-4k7tc"
Jul 30 02:43:48.682495 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:48.682473    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4r4p\" (UniqueName: \"kubernetes.io/projected/4b1585e2-ab06-4fc4-ad31-c3cf9912ea2b-kube-api-access-f4r4p\") pod \"maas-controller-6899b54cb8-4k7tc\" (UID: \"4b1585e2-ab06-4fc4-ad31-c3cf9912ea2b\") " pod="opendatahub/maas-controller-6899b54cb8-4k7tc"
Jul 30 02:43:48.784703 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:48.784612    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6899b54cb8-4k7tc"
Jul 30 02:43:48.802683 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:48.802655    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c77f4d6c-0fce-4885-8a2f-00c5d946ffa8" path="/var/lib/kubelet/pods/c77f4d6c-0fce-4885-8a2f-00c5d946ffa8/volumes"
Jul 30 02:43:48.919339 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:48.919306    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6899b54cb8-4k7tc"]
Jul 30 02:43:48.920849 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:43:48.920820    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1585e2_ab06_4fc4_ad31_c3cf9912ea2b.slice/crio-8820d7f07ae5070384642fc36c23bdaff4d4f1b024b246dbe9eab874180d9537 WatchSource:0}: Error finding container 8820d7f07ae5070384642fc36c23bdaff4d4f1b024b246dbe9eab874180d9537: Status 404 returned error can't find the container with id 8820d7f07ae5070384642fc36c23bdaff4d4f1b024b246dbe9eab874180d9537
Jul 30 02:43:49.674433 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:49.674339    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6899b54cb8-4k7tc" event={"ID":"4b1585e2-ab06-4fc4-ad31-c3cf9912ea2b","Type":"ContainerStarted","Data":"ca37b32a67e0fbe6e86eb337e4303d9038f7b70343fa1f5888ef701cdedec4d2"}
Jul 30 02:43:49.674433 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:49.674407    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6899b54cb8-4k7tc" event={"ID":"4b1585e2-ab06-4fc4-ad31-c3cf9912ea2b","Type":"ContainerStarted","Data":"8820d7f07ae5070384642fc36c23bdaff4d4f1b024b246dbe9eab874180d9537"}
Jul 30 02:43:49.674865 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:49.674577    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6899b54cb8-4k7tc"
Jul 30 02:43:49.693179 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:43:49.693124    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6899b54cb8-4k7tc" podStartSLOduration=1.283842413 podStartE2EDuration="1.693108695s" totalImagesPullingTime="409.266282ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:43:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:43:49.690336074 +0000 UTC m=+1065.337119492" watchObservedRunningTime="2026-07-30 02:43:49.693108695 +0000 UTC m=+1065.339892128"
Jul 30 02:44:00.682593 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:44:00.682562    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6899b54cb8-4k7tc"
Jul 30 02:45:00.145383 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:45:00.145345    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756325-dbq98"]
Jul 30 02:45:00.149060 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:45:00.149042    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756325-dbq98"
Jul 30 02:45:00.152396 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:45:00.152364    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2npk7\""
Jul 30 02:45:00.158196 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:45:00.158159    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756325-dbq98"
Jul 30 02:45:00.170501 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:45:00.166076    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756325-dbq98"]
Jul 30 02:45:00.297677 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:45:00.297649    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756325-dbq98"]
Jul 30 02:45:00.298754 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:45:00.298732    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ede0bf8_35cf_4b68_aa08_c9db0df578e3.slice/crio-805d1552fa0760863dc9ca149bdd745ef74a83f9e2b71439ccc228725d347df5 WatchSource:0}: Error finding container 805d1552fa0760863dc9ca149bdd745ef74a83f9e2b71439ccc228725d347df5: Status 404 returned error can't find the container with id 805d1552fa0760863dc9ca149bdd745ef74a83f9e2b71439ccc228725d347df5
Jul 30 02:45:01.060622 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:45:01.060587    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756325-dbq98" event={"ID":"4ede0bf8-35cf-4b68-aa08-c9db0df578e3","Type":"ContainerStarted","Data":"805d1552fa0760863dc9ca149bdd745ef74a83f9e2b71439ccc228725d347df5"}
Jul 30 02:45:03.074787 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:45:03.074745    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756325-dbq98" event={"ID":"4ede0bf8-35cf-4b68-aa08-c9db0df578e3","Type":"ContainerStarted","Data":"80939dc6c1c854e702c670222ba0bbf29e814cd860b0935df121fb284d43b727"}
Jul 30 02:45:03.108245 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:45:03.108192    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756325-dbq98" podStartSLOduration=0.74464021 podStartE2EDuration="3.108160118s" totalImagesPullingTime="2.363519908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:45:03.1076766 +0000 UTC m=+1138.754460005" watchObservedRunningTime="2026-07-30 02:45:03.108160118 +0000 UTC m=+1138.754943720"
Jul 30 02:45:04.081445 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:45:04.081406    2565 generic.go:356] "Generic (PLEG): container finished" podID="4ede0bf8-35cf-4b68-aa08-c9db0df578e3" containerID="80939dc6c1c854e702c670222ba0bbf29e814cd860b0935df121fb284d43b727" exitCode=0
Jul 30 02:45:04.081829 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:45:04.081448    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756325-dbq98" event={"ID":"4ede0bf8-35cf-4b68-aa08-c9db0df578e3","Type":"ContainerDied","Data":"80939dc6c1c854e702c670222ba0bbf29e814cd860b0935df121fb284d43b727"}
Jul 30 02:45:05.221187 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:45:05.221143    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756325-dbq98"
Jul 30 02:45:06.093306 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:45:06.093283    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756325-dbq98"
Jul 30 02:45:06.093606 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:45:06.093274    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756325-dbq98" event={"ID":"4ede0bf8-35cf-4b68-aa08-c9db0df578e3","Type":"ContainerDied","Data":"805d1552fa0760863dc9ca149bdd745ef74a83f9e2b71439ccc228725d347df5"}
Jul 30 02:45:06.093606 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:45:06.093397    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="805d1552fa0760863dc9ca149bdd745ef74a83f9e2b71439ccc228725d347df5"
Jul 30 02:49:03.192692 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:49:03.192647    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz"]
Jul 30 02:49:03.193238 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:49:03.192859    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz" podUID="985a4301-e06d-4ab1-b967-8d2eaff835f0" containerName="manager" containerID="cri-o://b4877a0e384273257c0cbc4c6a0c24192c2315e4e0b3feae7add3258e349ef3f" gracePeriod=10
Jul 30 02:49:03.372344 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:49:03.372310    2565 generic.go:356] "Generic (PLEG): container finished" podID="985a4301-e06d-4ab1-b967-8d2eaff835f0" containerID="b4877a0e384273257c0cbc4c6a0c24192c2315e4e0b3feae7add3258e349ef3f" exitCode=0
Jul 30 02:49:03.372472 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:49:03.372369    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz" event={"ID":"985a4301-e06d-4ab1-b967-8d2eaff835f0","Type":"ContainerDied","Data":"b4877a0e384273257c0cbc4c6a0c24192c2315e4e0b3feae7add3258e349ef3f"}
Jul 30 02:49:03.453464 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:49:03.453407    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz"
Jul 30 02:49:03.556755 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:49:03.556723    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/985a4301-e06d-4ab1-b967-8d2eaff835f0-extensions-socket-volume\") pod \"985a4301-e06d-4ab1-b967-8d2eaff835f0\" (UID: \"985a4301-e06d-4ab1-b967-8d2eaff835f0\") "
Jul 30 02:49:03.556915 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:49:03.556788    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnxg5\" (UniqueName: \"kubernetes.io/projected/985a4301-e06d-4ab1-b967-8d2eaff835f0-kube-api-access-nnxg5\") pod \"985a4301-e06d-4ab1-b967-8d2eaff835f0\" (UID: \"985a4301-e06d-4ab1-b967-8d2eaff835f0\") "
Jul 30 02:49:03.557088 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:49:03.557066    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/985a4301-e06d-4ab1-b967-8d2eaff835f0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "985a4301-e06d-4ab1-b967-8d2eaff835f0" (UID: "985a4301-e06d-4ab1-b967-8d2eaff835f0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:49:03.558878 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:49:03.558853    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985a4301-e06d-4ab1-b967-8d2eaff835f0-kube-api-access-nnxg5" (OuterVolumeSpecName: "kube-api-access-nnxg5") pod "985a4301-e06d-4ab1-b967-8d2eaff835f0" (UID: "985a4301-e06d-4ab1-b967-8d2eaff835f0"). InnerVolumeSpecName "kube-api-access-nnxg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:49:03.657613 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:49:03.657583    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnxg5\" (UniqueName: \"kubernetes.io/projected/985a4301-e06d-4ab1-b967-8d2eaff835f0-kube-api-access-nnxg5\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:49:03.657613 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:49:03.657608    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/985a4301-e06d-4ab1-b967-8d2eaff835f0-extensions-socket-volume\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:49:04.379228 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:49:04.379198    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz"
Jul 30 02:49:04.379228 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:49:04.379209    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz" event={"ID":"985a4301-e06d-4ab1-b967-8d2eaff835f0","Type":"ContainerDied","Data":"13dcb0556a95bd54a1ece5e8c2e2119d85249773b31bd03ec5b9af5175173dc2"}
Jul 30 02:49:04.379715 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:49:04.379273    2565 scope.go:117] "RemoveContainer" containerID="b4877a0e384273257c0cbc4c6a0c24192c2315e4e0b3feae7add3258e349ef3f"
Jul 30 02:49:04.402239 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:49:04.402217    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz"]
Jul 30 02:49:04.407513 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:49:04.407489    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pkthz"]
Jul 30 02:49:04.802595 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:49:04.802564    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="985a4301-e06d-4ab1-b967-8d2eaff835f0" path="/var/lib/kubelet/pods/985a4301-e06d-4ab1-b967-8d2eaff835f0/volumes"
Jul 30 02:50:03.856912 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:03.856820    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz"]
Jul 30 02:50:03.857584 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:03.857419    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="985a4301-e06d-4ab1-b967-8d2eaff835f0" containerName="manager"
Jul 30 02:50:03.857584 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:03.857439    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="985a4301-e06d-4ab1-b967-8d2eaff835f0" containerName="manager"
Jul 30 02:50:03.857584 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:03.857455    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ede0bf8-35cf-4b68-aa08-c9db0df578e3" containerName="cleanup"
Jul 30 02:50:03.857584 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:03.857463    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ede0bf8-35cf-4b68-aa08-c9db0df578e3" containerName="cleanup"
Jul 30 02:50:03.857584 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:03.857549    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ede0bf8-35cf-4b68-aa08-c9db0df578e3" containerName="cleanup"
Jul 30 02:50:03.857584 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:03.857562    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="985a4301-e06d-4ab1-b967-8d2eaff835f0" containerName="manager"
Jul 30 02:50:03.860627 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:03.860607    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz"
Jul 30 02:50:03.863798 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:03.863780    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kgss5\""
Jul 30 02:50:03.872079 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:03.872054    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz"]
Jul 30 02:50:04.005513 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:04.005485    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e0030d3-06ab-49b6-b3d0-418364956512-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-47gkz\" (UID: \"4e0030d3-06ab-49b6-b3d0-418364956512\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz"
Jul 30 02:50:04.005677 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:04.005564    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6prgj\" (UniqueName: \"kubernetes.io/projected/4e0030d3-06ab-49b6-b3d0-418364956512-kube-api-access-6prgj\") pod \"kuadrant-operator-controller-manager-86878c769f-47gkz\" (UID: \"4e0030d3-06ab-49b6-b3d0-418364956512\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz"
Jul 30 02:50:04.106327 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:04.106296    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6prgj\" (UniqueName: \"kubernetes.io/projected/4e0030d3-06ab-49b6-b3d0-418364956512-kube-api-access-6prgj\") pod \"kuadrant-operator-controller-manager-86878c769f-47gkz\" (UID: \"4e0030d3-06ab-49b6-b3d0-418364956512\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz"
Jul 30 02:50:04.106487 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:04.106359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e0030d3-06ab-49b6-b3d0-418364956512-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-47gkz\" (UID: \"4e0030d3-06ab-49b6-b3d0-418364956512\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz"
Jul 30 02:50:04.106727 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:04.106711    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e0030d3-06ab-49b6-b3d0-418364956512-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-47gkz\" (UID: \"4e0030d3-06ab-49b6-b3d0-418364956512\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz"
Jul 30 02:50:04.128111 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:04.128034    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6prgj\" (UniqueName: \"kubernetes.io/projected/4e0030d3-06ab-49b6-b3d0-418364956512-kube-api-access-6prgj\") pod \"kuadrant-operator-controller-manager-86878c769f-47gkz\" (UID: \"4e0030d3-06ab-49b6-b3d0-418364956512\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz"
Jul 30 02:50:04.172007 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:04.171975    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz"
Jul 30 02:50:04.321560 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:04.321532    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz"]
Jul 30 02:50:04.323482 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:50:04.323454    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0030d3_06ab_49b6_b3d0_418364956512.slice/crio-82226b88b75ea5a35a0e57b9e6089dd838fca3ee079febbe2ee6ea8755e6abd5 WatchSource:0}: Error finding container 82226b88b75ea5a35a0e57b9e6089dd838fca3ee079febbe2ee6ea8755e6abd5: Status 404 returned error can't find the container with id 82226b88b75ea5a35a0e57b9e6089dd838fca3ee079febbe2ee6ea8755e6abd5
Jul 30 02:50:04.689519 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:04.689440    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz" event={"ID":"4e0030d3-06ab-49b6-b3d0-418364956512","Type":"ContainerStarted","Data":"cd42da0bb2d02d57547d2d79256e9f2ee75d457b1004c07fef7767d1c459f634"}
Jul 30 02:50:04.689519 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:04.689492    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz" event={"ID":"4e0030d3-06ab-49b6-b3d0-418364956512","Type":"ContainerStarted","Data":"82226b88b75ea5a35a0e57b9e6089dd838fca3ee079febbe2ee6ea8755e6abd5"}
Jul 30 02:50:04.689712 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:04.689523    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz"
Jul 30 02:50:04.715209 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:04.715139    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz" podStartSLOduration=1.715124432 podStartE2EDuration="1.715124432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:50:04.713635593 +0000 UTC m=+1440.360418998" watchObservedRunningTime="2026-07-30 02:50:04.715124432 +0000 UTC m=+1440.361907834"
Jul 30 02:50:15.696537 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:50:15.696505    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz"
Jul 30 02:57:37.938783 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:37.938752    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"]
Jul 30 02:57:37.942320 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:37.942304    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:37.945369 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:37.945349    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d054fc9a-openshift-default-dockercfg-24j5g\""
Jul 30 02:57:37.965606 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:37.965583    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"]
Jul 30 02:57:38.051894 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.051856    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-workload-certs\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.051894 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.051892    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-workload-socket\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.052122 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.051911    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-envoy\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.052122 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.051942    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-podinfo\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.052122 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.051979    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th5g9\" (UniqueName: \"kubernetes.io/projected/36a847ff-2a8a-42c3-9c88-302f2fded881-kube-api-access-th5g9\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.052122 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.052037    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a847ff-2a8a-42c3-9c88-302f2fded881-istiod-ca-cert\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.052122 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.052069    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-credential-socket\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.052122 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.052103    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-token\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.052363 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.052133    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-data\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.153372 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.153341    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-data\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.153568 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.153403    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-workload-certs\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.153568 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.153431    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-workload-socket\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.153568 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.153453    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-envoy\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.153568 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.153488    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-podinfo\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.153797 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.153615    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th5g9\" (UniqueName: \"kubernetes.io/projected/36a847ff-2a8a-42c3-9c88-302f2fded881-kube-api-access-th5g9\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.153797 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.153703    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a847ff-2a8a-42c3-9c88-302f2fded881-istiod-ca-cert\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.153797 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.153740    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-credential-socket\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.153797 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.153786    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-workload-certs\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.153797 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.153791    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-token\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.154059 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.153833    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-workload-socket\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.154059 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.153704    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-data\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.154059 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.154031    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-credential-socket\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.154405 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.154354    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a847ff-2a8a-42c3-9c88-302f2fded881-istiod-ca-cert\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.155850 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.155829    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-envoy\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.156051 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.156034    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-podinfo\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.164056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.164037    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-token\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.164147 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.164033    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th5g9\" (UniqueName: \"kubernetes.io/projected/36a847ff-2a8a-42c3-9c88-302f2fded881-kube-api-access-th5g9\") pod \"e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") " pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.252906 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.252813    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:38.393103 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.393051    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"]
Jul 30 02:57:38.394056 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:57:38.394031    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a847ff_2a8a_42c3_9c88_302f2fded881.slice/crio-7b3bb6e389cce8d8d997fedcd7b063bf940f77a0269b51d56e813635302e7d95 WatchSource:0}: Error finding container 7b3bb6e389cce8d8d997fedcd7b063bf940f77a0269b51d56e813635302e7d95: Status 404 returned error can't find the container with id 7b3bb6e389cce8d8d997fedcd7b063bf940f77a0269b51d56e813635302e7d95
Jul 30 02:57:38.396313 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.396274    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:57:38.396380 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.396366    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:57:38.396427 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:38.396411    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:57:39.158075 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:39.158044    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj" event={"ID":"36a847ff-2a8a-42c3-9c88-302f2fded881","Type":"ContainerStarted","Data":"658b7fe62e944eb1cff67a625dcb596d412d1baf635fde5c10eb909abae80d41"}
Jul 30 02:57:39.158486 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:39.158095    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj" event={"ID":"36a847ff-2a8a-42c3-9c88-302f2fded881","Type":"ContainerStarted","Data":"7b3bb6e389cce8d8d997fedcd7b063bf940f77a0269b51d56e813635302e7d95"}
Jul 30 02:57:39.184926 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:39.184876    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj" podStartSLOduration=2.184858976 podStartE2EDuration="2.184858976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:57:39.183392306 +0000 UTC m=+1894.830175711" watchObservedRunningTime="2026-07-30 02:57:39.184858976 +0000 UTC m=+1894.831642380"
Jul 30 02:57:39.253786 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:39.253748    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:39.258319 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:39.258296    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:40.164030 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:40.163999    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:40.164938 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:40.164921    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:40.931828 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:40.931792    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"]
Jul 30 02:57:40.935766 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:40.935742    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"
Jul 30 02:57:40.939686 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:40.939664    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d054fc9a-serving-cert\""
Jul 30 02:57:40.948221 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:40.948198    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"]
Jul 30 02:57:40.982130 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:40.982096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtk2x\" (UniqueName: \"kubernetes.io/projected/748363c8-9b89-459c-8f96-e226b8782854-kube-api-access-wtk2x\") pod \"maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn\" (UID: \"748363c8-9b89-459c-8f96-e226b8782854\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"
Jul 30 02:57:40.982325 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:40.982216    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/748363c8-9b89-459c-8f96-e226b8782854-openshift-service-ca\") pod \"maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn\" (UID: \"748363c8-9b89-459c-8f96-e226b8782854\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"
Jul 30 02:57:40.982325 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:40.982257    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/748363c8-9b89-459c-8f96-e226b8782854-maas-api-tls\") pod \"maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn\" (UID: \"748363c8-9b89-459c-8f96-e226b8782854\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"
Jul 30 02:57:41.083078 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.083042    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtk2x\" (UniqueName: \"kubernetes.io/projected/748363c8-9b89-459c-8f96-e226b8782854-kube-api-access-wtk2x\") pod \"maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn\" (UID: \"748363c8-9b89-459c-8f96-e226b8782854\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"
Jul 30 02:57:41.083279 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.083114    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/748363c8-9b89-459c-8f96-e226b8782854-openshift-service-ca\") pod \"maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn\" (UID: \"748363c8-9b89-459c-8f96-e226b8782854\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"
Jul 30 02:57:41.083279 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.083149    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/748363c8-9b89-459c-8f96-e226b8782854-maas-api-tls\") pod \"maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn\" (UID: \"748363c8-9b89-459c-8f96-e226b8782854\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"
Jul 30 02:57:41.083357 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:57:41.083316    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-serving-cert: secret "maas-api-e2e-ait-d054fc9a-serving-cert" not found
Jul 30 02:57:41.083408 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:57:41.083396    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/748363c8-9b89-459c-8f96-e226b8782854-maas-api-tls podName:748363c8-9b89-459c-8f96-e226b8782854 nodeName:}" failed. No retries permitted until 2026-07-30 02:57:41.583373377 +0000 UTC m=+1897.230156772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/748363c8-9b89-459c-8f96-e226b8782854-maas-api-tls") pod "maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn" (UID: "748363c8-9b89-459c-8f96-e226b8782854") : secret "maas-api-e2e-ait-d054fc9a-serving-cert" not found
Jul 30 02:57:41.083781 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.083748    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/748363c8-9b89-459c-8f96-e226b8782854-openshift-service-ca\") pod \"maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn\" (UID: \"748363c8-9b89-459c-8f96-e226b8782854\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"
Jul 30 02:57:41.093141 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.093112    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtk2x\" (UniqueName: \"kubernetes.io/projected/748363c8-9b89-459c-8f96-e226b8782854-kube-api-access-wtk2x\") pod \"maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn\" (UID: \"748363c8-9b89-459c-8f96-e226b8782854\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"
Jul 30 02:57:41.588465 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.588431    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/748363c8-9b89-459c-8f96-e226b8782854-maas-api-tls\") pod \"maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn\" (UID: \"748363c8-9b89-459c-8f96-e226b8782854\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"
Jul 30 02:57:41.590977 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.590958    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/748363c8-9b89-459c-8f96-e226b8782854-maas-api-tls\") pod \"maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn\" (UID: \"748363c8-9b89-459c-8f96-e226b8782854\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"
Jul 30 02:57:41.790092 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.790041    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj"]
Jul 30 02:57:41.793886 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.793868    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj"
Jul 30 02:57:41.796787 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.796767    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d054fc9a-dockercfg-sksdt\""
Jul 30 02:57:41.796787 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.796775    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d054fc9a\""
Jul 30 02:57:41.803598 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.803571    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj"]
Jul 30 02:57:41.834666 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.834621    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t"]
Jul 30 02:57:41.838947 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.838870    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t"
Jul 30 02:57:41.847859 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.847832    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t"]
Jul 30 02:57:41.851869 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.848048    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"
Jul 30 02:57:41.891632 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.891133    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5791e3fc-4f4c-4d1d-87dd-e33ba303d001-plugins-config-volume\") pod \"payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t\" (UID: \"5791e3fc-4f4c-4d1d-87dd-e33ba303d001\") " pod="openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t"
Jul 30 02:57:41.891632 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.891189    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bb8z\" (UniqueName: \"kubernetes.io/projected/5791e3fc-4f4c-4d1d-87dd-e33ba303d001-kube-api-access-4bb8z\") pod \"payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t\" (UID: \"5791e3fc-4f4c-4d1d-87dd-e33ba303d001\") " pod="openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t"
Jul 30 02:57:41.891632 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.891235    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f117cabe-6077-47f1-ad63-8d0f18bd610f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj\" (UID: \"f117cabe-6077-47f1-ad63-8d0f18bd610f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj"
Jul 30 02:57:41.891632 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.891264    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs552\" (UniqueName: \"kubernetes.io/projected/f117cabe-6077-47f1-ad63-8d0f18bd610f-kube-api-access-vs552\") pod \"payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj\" (UID: \"f117cabe-6077-47f1-ad63-8d0f18bd610f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj"
Jul 30 02:57:41.992782 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.992748    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f117cabe-6077-47f1-ad63-8d0f18bd610f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj\" (UID: \"f117cabe-6077-47f1-ad63-8d0f18bd610f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj"
Jul 30 02:57:41.992782 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.992792    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs552\" (UniqueName: \"kubernetes.io/projected/f117cabe-6077-47f1-ad63-8d0f18bd610f-kube-api-access-vs552\") pod \"payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj\" (UID: \"f117cabe-6077-47f1-ad63-8d0f18bd610f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj"
Jul 30 02:57:41.993041 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.992908    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5791e3fc-4f4c-4d1d-87dd-e33ba303d001-plugins-config-volume\") pod \"payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t\" (UID: \"5791e3fc-4f4c-4d1d-87dd-e33ba303d001\") " pod="openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t"
Jul 30 02:57:41.993041 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.992947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bb8z\" (UniqueName: \"kubernetes.io/projected/5791e3fc-4f4c-4d1d-87dd-e33ba303d001-kube-api-access-4bb8z\") pod \"payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t\" (UID: \"5791e3fc-4f4c-4d1d-87dd-e33ba303d001\") " pod="openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t"
Jul 30 02:57:41.993531 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.993512    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f117cabe-6077-47f1-ad63-8d0f18bd610f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj\" (UID: \"f117cabe-6077-47f1-ad63-8d0f18bd610f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj"
Jul 30 02:57:41.993597 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:41.993578    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5791e3fc-4f4c-4d1d-87dd-e33ba303d001-plugins-config-volume\") pod \"payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t\" (UID: \"5791e3fc-4f4c-4d1d-87dd-e33ba303d001\") " pod="openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t"
Jul 30 02:57:42.004035 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:42.004005    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"]
Jul 30 02:57:42.004788 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:42.004765    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs552\" (UniqueName: \"kubernetes.io/projected/f117cabe-6077-47f1-ad63-8d0f18bd610f-kube-api-access-vs552\") pod \"payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj\" (UID: \"f117cabe-6077-47f1-ad63-8d0f18bd610f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj"
Jul 30 02:57:42.004906 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:42.004887    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bb8z\" (UniqueName: \"kubernetes.io/projected/5791e3fc-4f4c-4d1d-87dd-e33ba303d001-kube-api-access-4bb8z\") pod \"payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t\" (UID: \"5791e3fc-4f4c-4d1d-87dd-e33ba303d001\") " pod="openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t"
Jul 30 02:57:42.005540 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:57:42.005512    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod748363c8_9b89_459c_8f96_e226b8782854.slice/crio-350fd88597894239284551b85946410c1bdfcadbc67573e914bb664ba1f762f6 WatchSource:0}: Error finding container 350fd88597894239284551b85946410c1bdfcadbc67573e914bb664ba1f762f6: Status 404 returned error can't find the container with id 350fd88597894239284551b85946410c1bdfcadbc67573e914bb664ba1f762f6
Jul 30 02:57:42.007108 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:42.007086    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 02:57:42.106519 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:42.106432    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj"
Jul 30 02:57:42.158147 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:42.158085    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t"
Jul 30 02:57:42.179098 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:42.179060    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn" event={"ID":"748363c8-9b89-459c-8f96-e226b8782854","Type":"ContainerStarted","Data":"350fd88597894239284551b85946410c1bdfcadbc67573e914bb664ba1f762f6"}
Jul 30 02:57:42.262460 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:42.262428    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj"]
Jul 30 02:57:42.263091 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:57:42.263058    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf117cabe_6077_47f1_ad63_8d0f18bd610f.slice/crio-a44344d587c7e790b83ca11d11809e058afb0bd3dac24589935402a53d0877d6 WatchSource:0}: Error finding container a44344d587c7e790b83ca11d11809e058afb0bd3dac24589935402a53d0877d6: Status 404 returned error can't find the container with id a44344d587c7e790b83ca11d11809e058afb0bd3dac24589935402a53d0877d6
Jul 30 02:57:42.318300 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:42.318274    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t"]
Jul 30 02:57:42.320118 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:57:42.320094    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5791e3fc_4f4c_4d1d_87dd_e33ba303d001.slice/crio-480dd82d21e59f51f89fa26ef5157823614bb1162f21ce818bc77aeac66ff341 WatchSource:0}: Error finding container 480dd82d21e59f51f89fa26ef5157823614bb1162f21ce818bc77aeac66ff341: Status 404 returned error can't find the container with id 480dd82d21e59f51f89fa26ef5157823614bb1162f21ce818bc77aeac66ff341
Jul 30 02:57:43.188762 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:43.188720    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t" event={"ID":"5791e3fc-4f4c-4d1d-87dd-e33ba303d001","Type":"ContainerStarted","Data":"9b4a29f3a2f8387c99478f9b758205365135bb81b07a79d94484c64def0ac9a3"}
Jul 30 02:57:43.189276 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:43.188800    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t"
Jul 30 02:57:43.189276 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:43.188827    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t" event={"ID":"5791e3fc-4f4c-4d1d-87dd-e33ba303d001","Type":"ContainerStarted","Data":"480dd82d21e59f51f89fa26ef5157823614bb1162f21ce818bc77aeac66ff341"}
Jul 30 02:57:43.190699 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:43.190659    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn" event={"ID":"748363c8-9b89-459c-8f96-e226b8782854","Type":"ContainerStarted","Data":"fbcef63af7aebb4eae65825a336e9873f660f10633e4bf80153d843d687cd88f"}
Jul 30 02:57:43.191078 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:43.190724    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"
Jul 30 02:57:43.192249 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:43.192227    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj" event={"ID":"f117cabe-6077-47f1-ad63-8d0f18bd610f","Type":"ContainerStarted","Data":"ccaec3f6b4b21dfd3d142e5b0cffa4217cbc5b91b5fd381031ea45740d0a6d9a"}
Jul 30 02:57:43.192362 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:43.192266    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj" event={"ID":"f117cabe-6077-47f1-ad63-8d0f18bd610f","Type":"ContainerStarted","Data":"a44344d587c7e790b83ca11d11809e058afb0bd3dac24589935402a53d0877d6"}
Jul 30 02:57:43.192406 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:43.192370    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj"
Jul 30 02:57:43.207193 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:43.207135    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t" podStartSLOduration=2.207120377 podStartE2EDuration="2.207120377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:57:43.204971788 +0000 UTC m=+1898.851755192" watchObservedRunningTime="2026-07-30 02:57:43.207120377 +0000 UTC m=+1898.853903760"
Jul 30 02:57:43.222977 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:43.222934    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn" podStartSLOduration=2.7196028979999998 podStartE2EDuration="3.222917694s" totalImagesPullingTime="503.314796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:57:43.221464127 +0000 UTC m=+1898.868247532" watchObservedRunningTime="2026-07-30 02:57:43.222917694 +0000 UTC m=+1898.869701098"
Jul 30 02:57:43.238758 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:43.238710    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj" podStartSLOduration=2.238695441 podStartE2EDuration="2.238695441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:57:43.23600153 +0000 UTC m=+1898.882784937" watchObservedRunningTime="2026-07-30 02:57:43.238695441 +0000 UTC m=+1898.885478877"
Jul 30 02:57:49.207837 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:49.207799    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"
Jul 30 02:57:54.204831 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:54.204798    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t"
Jul 30 02:57:54.205332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:54.204995    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj"
Jul 30 02:57:55.809716 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:55.808792    2565 patch_prober.go:28] interesting pod/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": read tcp 10.134.0.2:39814->10.134.0.54:15021: read: connection reset by peer" start-of-body=
Jul 30 02:57:55.809716 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:55.808855    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj" podUID="36a847ff-2a8a-42c3-9c88-302f2fded881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": read tcp 10.134.0.2:39814->10.134.0.54:15021: read: connection reset by peer"
Jul 30 02:57:56.271596 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:56.271511    2565 generic.go:356] "Generic (PLEG): container finished" podID="36a847ff-2a8a-42c3-9c88-302f2fded881" containerID="658b7fe62e944eb1cff67a625dcb596d412d1baf635fde5c10eb909abae80d41" exitCode=0
Jul 30 02:57:56.271742 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:56.271586    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj" event={"ID":"36a847ff-2a8a-42c3-9c88-302f2fded881","Type":"ContainerDied","Data":"658b7fe62e944eb1cff67a625dcb596d412d1baf635fde5c10eb909abae80d41"}
Jul 30 02:57:56.271988 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:56.271973    2565 scope.go:117] "RemoveContainer" containerID="658b7fe62e944eb1cff67a625dcb596d412d1baf635fde5c10eb909abae80d41"
Jul 30 02:57:56.272999 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:56.272970    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:57:56.273056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:56.273042    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:57:56.273104 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:56.273079    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:57:57.283271 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:57.283224    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj" event={"ID":"36a847ff-2a8a-42c3-9c88-302f2fded881","Type":"ContainerStarted","Data":"9e927b920d240855ea823cdce14e4c9fc9f007804d43d131a2a1e0926ed22958"}
Jul 30 02:57:58.254007 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:58.253969    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:57:58.254231 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:58.254203    2565 patch_prober.go:28] interesting pod/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Jul 30 02:57:58.254364 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:58.254256    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj" podUID="36a847ff-2a8a-42c3-9c88-302f2fded881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Jul 30 02:57:59.253695 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:59.253654    2565 patch_prober.go:28] interesting pod/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Jul 30 02:57:59.254100 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:57:59.253721    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj" podUID="36a847ff-2a8a-42c3-9c88-302f2fded881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Jul 30 02:58:00.253904 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:00.253865    2565 patch_prober.go:28] interesting pod/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Jul 30 02:58:00.254333 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:00.253923    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj" podUID="36a847ff-2a8a-42c3-9c88-302f2fded881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Jul 30 02:58:01.256964 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:01.256930    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:58:01.257472 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:01.257365    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:58:01.258050 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:01.258032    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:58:06.065863 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:06.065827    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d054fc9a-zcq2w"]
Jul 30 02:58:06.070093 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:06.070073    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d054fc9a-zcq2w"
Jul 30 02:58:06.073043 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:06.073023    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2npk7\""
Jul 30 02:58:06.078360 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:06.078334    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d054fc9a-zcq2w"]
Jul 30 02:58:06.226117 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:06.226080    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/023ee300-8443-4cd8-aaee-4cb2ed21f6ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d054fc9a-zcq2w\" (UID: \"023ee300-8443-4cd8-aaee-4cb2ed21f6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d054fc9a-zcq2w"
Jul 30 02:58:06.327497 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:06.327405    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/023ee300-8443-4cd8-aaee-4cb2ed21f6ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d054fc9a-zcq2w\" (UID: \"023ee300-8443-4cd8-aaee-4cb2ed21f6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d054fc9a-zcq2w"
Jul 30 02:58:06.328010 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:06.327991    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/023ee300-8443-4cd8-aaee-4cb2ed21f6ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d054fc9a-zcq2w\" (UID: \"023ee300-8443-4cd8-aaee-4cb2ed21f6ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d054fc9a-zcq2w"
Jul 30 02:58:06.384458 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:06.384424    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d054fc9a-zcq2w"
Jul 30 02:58:06.524471 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:06.524439    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d054fc9a-zcq2w"]
Jul 30 02:58:06.526060 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:58:06.526032    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023ee300_8443_4cd8_aaee_4cb2ed21f6ff.slice/crio-5f55a48dd41aae1fff8f02ed9a5a1bf98dad7fc2dae592c9f5e412e67d1d041f WatchSource:0}: Error finding container 5f55a48dd41aae1fff8f02ed9a5a1bf98dad7fc2dae592c9f5e412e67d1d041f: Status 404 returned error can't find the container with id 5f55a48dd41aae1fff8f02ed9a5a1bf98dad7fc2dae592c9f5e412e67d1d041f
Jul 30 02:58:07.351787 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:07.351753    2565 generic.go:356] "Generic (PLEG): container finished" podID="023ee300-8443-4cd8-aaee-4cb2ed21f6ff" containerID="4af8c98d6aee1ad5338622d25c7789ec215746c2ff93b2170eeb8d06380956e6" exitCode=0
Jul 30 02:58:07.352150 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:07.351829    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d054fc9a-zcq2w" event={"ID":"023ee300-8443-4cd8-aaee-4cb2ed21f6ff","Type":"ContainerDied","Data":"4af8c98d6aee1ad5338622d25c7789ec215746c2ff93b2170eeb8d06380956e6"}
Jul 30 02:58:07.352150 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:07.351866    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d054fc9a-zcq2w" event={"ID":"023ee300-8443-4cd8-aaee-4cb2ed21f6ff","Type":"ContainerStarted","Data":"5f55a48dd41aae1fff8f02ed9a5a1bf98dad7fc2dae592c9f5e412e67d1d041f"}
Jul 30 02:58:08.489471 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:08.489443    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d054fc9a-zcq2w"
Jul 30 02:58:08.646968 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:08.646874    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/023ee300-8443-4cd8-aaee-4cb2ed21f6ff-maas-api-service-ca\") pod \"023ee300-8443-4cd8-aaee-4cb2ed21f6ff\" (UID: \"023ee300-8443-4cd8-aaee-4cb2ed21f6ff\") "
Jul 30 02:58:08.647255 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:08.647231    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023ee300-8443-4cd8-aaee-4cb2ed21f6ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "023ee300-8443-4cd8-aaee-4cb2ed21f6ff" (UID: "023ee300-8443-4cd8-aaee-4cb2ed21f6ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 02:58:08.748117 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:08.748089    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/023ee300-8443-4cd8-aaee-4cb2ed21f6ff-maas-api-service-ca\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:58:09.363576 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:09.363543    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d054fc9a-zcq2w"
Jul 30 02:58:09.363743 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:09.363580    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d054fc9a-zcq2w" event={"ID":"023ee300-8443-4cd8-aaee-4cb2ed21f6ff","Type":"ContainerDied","Data":"5f55a48dd41aae1fff8f02ed9a5a1bf98dad7fc2dae592c9f5e412e67d1d041f"}
Jul 30 02:58:09.363743 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:09.363629    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f55a48dd41aae1fff8f02ed9a5a1bf98dad7fc2dae592c9f5e412e67d1d041f"
Jul 30 02:58:16.469025 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.468989    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"]
Jul 30 02:58:16.469470 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.469236    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn" podUID="748363c8-9b89-459c-8f96-e226b8782854" containerName="maas-api" containerID="cri-o://fbcef63af7aebb4eae65825a336e9873f660f10633e4bf80153d843d687cd88f" gracePeriod=30
Jul 30 02:58:16.512348 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.512314    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t"]
Jul 30 02:58:16.512599 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.512570    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t" podUID="5791e3fc-4f4c-4d1d-87dd-e33ba303d001" containerName="payload-processing" containerID="cri-o://9b4a29f3a2f8387c99478f9b758205365135bb81b07a79d94484c64def0ac9a3" gracePeriod=30
Jul 30 02:58:16.518928 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.518833    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj"]
Jul 30 02:58:16.519126 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.519100    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj" podUID="f117cabe-6077-47f1-ad63-8d0f18bd610f" containerName="payload-pre-processing" containerID="cri-o://ccaec3f6b4b21dfd3d142e5b0cffa4217cbc5b91b5fd381031ea45740d0a6d9a" gracePeriod=30
Jul 30 02:58:16.828057 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.828022    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"
Jul 30 02:58:16.843258 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.843234    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj"
Jul 30 02:58:16.851508 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.851483    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t"
Jul 30 02:58:16.921414 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.921377    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vs552\" (UniqueName: \"kubernetes.io/projected/f117cabe-6077-47f1-ad63-8d0f18bd610f-kube-api-access-vs552\") pod \"f117cabe-6077-47f1-ad63-8d0f18bd610f\" (UID: \"f117cabe-6077-47f1-ad63-8d0f18bd610f\") "
Jul 30 02:58:16.921414 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.921417    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f117cabe-6077-47f1-ad63-8d0f18bd610f-plugins-config-volume\") pod \"f117cabe-6077-47f1-ad63-8d0f18bd610f\" (UID: \"f117cabe-6077-47f1-ad63-8d0f18bd610f\") "
Jul 30 02:58:16.921644 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.921465    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/748363c8-9b89-459c-8f96-e226b8782854-openshift-service-ca\") pod \"748363c8-9b89-459c-8f96-e226b8782854\" (UID: \"748363c8-9b89-459c-8f96-e226b8782854\") "
Jul 30 02:58:16.921644 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.921497    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bb8z\" (UniqueName: \"kubernetes.io/projected/5791e3fc-4f4c-4d1d-87dd-e33ba303d001-kube-api-access-4bb8z\") pod \"5791e3fc-4f4c-4d1d-87dd-e33ba303d001\" (UID: \"5791e3fc-4f4c-4d1d-87dd-e33ba303d001\") "
Jul 30 02:58:16.921644 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.921536    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtk2x\" (UniqueName: \"kubernetes.io/projected/748363c8-9b89-459c-8f96-e226b8782854-kube-api-access-wtk2x\") pod \"748363c8-9b89-459c-8f96-e226b8782854\" (UID: \"748363c8-9b89-459c-8f96-e226b8782854\") "
Jul 30 02:58:16.921644 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.921568    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5791e3fc-4f4c-4d1d-87dd-e33ba303d001-plugins-config-volume\") pod \"5791e3fc-4f4c-4d1d-87dd-e33ba303d001\" (UID: \"5791e3fc-4f4c-4d1d-87dd-e33ba303d001\") "
Jul 30 02:58:16.921644 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.921602    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/748363c8-9b89-459c-8f96-e226b8782854-maas-api-tls\") pod \"748363c8-9b89-459c-8f96-e226b8782854\" (UID: \"748363c8-9b89-459c-8f96-e226b8782854\") "
Jul 30 02:58:16.921910 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.921813    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/748363c8-9b89-459c-8f96-e226b8782854-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "748363c8-9b89-459c-8f96-e226b8782854" (UID: "748363c8-9b89-459c-8f96-e226b8782854"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 02:58:16.921910 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.921859    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f117cabe-6077-47f1-ad63-8d0f18bd610f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f117cabe-6077-47f1-ad63-8d0f18bd610f" (UID: "f117cabe-6077-47f1-ad63-8d0f18bd610f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 02:58:16.922134 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.922101    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5791e3fc-4f4c-4d1d-87dd-e33ba303d001-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5791e3fc-4f4c-4d1d-87dd-e33ba303d001" (UID: "5791e3fc-4f4c-4d1d-87dd-e33ba303d001"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 02:58:16.922452 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.922137    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f117cabe-6077-47f1-ad63-8d0f18bd610f-plugins-config-volume\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:58:16.922452 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.922156    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/748363c8-9b89-459c-8f96-e226b8782854-openshift-service-ca\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:58:16.924035 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.924010    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748363c8-9b89-459c-8f96-e226b8782854-kube-api-access-wtk2x" (OuterVolumeSpecName: "kube-api-access-wtk2x") pod "748363c8-9b89-459c-8f96-e226b8782854" (UID: "748363c8-9b89-459c-8f96-e226b8782854"). InnerVolumeSpecName "kube-api-access-wtk2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:58:16.924139 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.924094    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f117cabe-6077-47f1-ad63-8d0f18bd610f-kube-api-access-vs552" (OuterVolumeSpecName: "kube-api-access-vs552") pod "f117cabe-6077-47f1-ad63-8d0f18bd610f" (UID: "f117cabe-6077-47f1-ad63-8d0f18bd610f"). InnerVolumeSpecName "kube-api-access-vs552". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:58:16.924323 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.924304    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/748363c8-9b89-459c-8f96-e226b8782854-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "748363c8-9b89-459c-8f96-e226b8782854" (UID: "748363c8-9b89-459c-8f96-e226b8782854"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 02:58:16.924379 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:16.924304    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5791e3fc-4f4c-4d1d-87dd-e33ba303d001-kube-api-access-4bb8z" (OuterVolumeSpecName: "kube-api-access-4bb8z") pod "5791e3fc-4f4c-4d1d-87dd-e33ba303d001" (UID: "5791e3fc-4f4c-4d1d-87dd-e33ba303d001"). InnerVolumeSpecName "kube-api-access-4bb8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:58:17.022827 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.022743    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs552\" (UniqueName: \"kubernetes.io/projected/f117cabe-6077-47f1-ad63-8d0f18bd610f-kube-api-access-vs552\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:58:17.022827 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.022771    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bb8z\" (UniqueName: \"kubernetes.io/projected/5791e3fc-4f4c-4d1d-87dd-e33ba303d001-kube-api-access-4bb8z\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:58:17.022827 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.022781    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtk2x\" (UniqueName: \"kubernetes.io/projected/748363c8-9b89-459c-8f96-e226b8782854-kube-api-access-wtk2x\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:58:17.022827 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.022789    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5791e3fc-4f4c-4d1d-87dd-e33ba303d001-plugins-config-volume\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:58:17.022827 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.022798    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/748363c8-9b89-459c-8f96-e226b8782854-maas-api-tls\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:58:17.410747 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.410710    2565 generic.go:356] "Generic (PLEG): container finished" podID="5791e3fc-4f4c-4d1d-87dd-e33ba303d001" containerID="9b4a29f3a2f8387c99478f9b758205365135bb81b07a79d94484c64def0ac9a3" exitCode=0
Jul 30 02:58:17.410918 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.410752    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t"
Jul 30 02:58:17.410918 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.410804    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t" event={"ID":"5791e3fc-4f4c-4d1d-87dd-e33ba303d001","Type":"ContainerDied","Data":"9b4a29f3a2f8387c99478f9b758205365135bb81b07a79d94484c64def0ac9a3"}
Jul 30 02:58:17.410918 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.410861    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t" event={"ID":"5791e3fc-4f4c-4d1d-87dd-e33ba303d001","Type":"ContainerDied","Data":"480dd82d21e59f51f89fa26ef5157823614bb1162f21ce818bc77aeac66ff341"}
Jul 30 02:58:17.410918 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.410892    2565 scope.go:117] "RemoveContainer" containerID="9b4a29f3a2f8387c99478f9b758205365135bb81b07a79d94484c64def0ac9a3"
Jul 30 02:58:17.412325 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.412300    2565 generic.go:356] "Generic (PLEG): container finished" podID="748363c8-9b89-459c-8f96-e226b8782854" containerID="fbcef63af7aebb4eae65825a336e9873f660f10633e4bf80153d843d687cd88f" exitCode=0
Jul 30 02:58:17.412617 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.412351    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"
Jul 30 02:58:17.412617 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.412387    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn" event={"ID":"748363c8-9b89-459c-8f96-e226b8782854","Type":"ContainerDied","Data":"fbcef63af7aebb4eae65825a336e9873f660f10633e4bf80153d843d687cd88f"}
Jul 30 02:58:17.412617 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.412559    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn" event={"ID":"748363c8-9b89-459c-8f96-e226b8782854","Type":"ContainerDied","Data":"350fd88597894239284551b85946410c1bdfcadbc67573e914bb664ba1f762f6"}
Jul 30 02:58:17.413817 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.413794    2565 generic.go:356] "Generic (PLEG): container finished" podID="f117cabe-6077-47f1-ad63-8d0f18bd610f" containerID="ccaec3f6b4b21dfd3d142e5b0cffa4217cbc5b91b5fd381031ea45740d0a6d9a" exitCode=0
Jul 30 02:58:17.413948 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.413839    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj" event={"ID":"f117cabe-6077-47f1-ad63-8d0f18bd610f","Type":"ContainerDied","Data":"ccaec3f6b4b21dfd3d142e5b0cffa4217cbc5b91b5fd381031ea45740d0a6d9a"}
Jul 30 02:58:17.414017 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.413993    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj" event={"ID":"f117cabe-6077-47f1-ad63-8d0f18bd610f","Type":"ContainerDied","Data":"a44344d587c7e790b83ca11d11809e058afb0bd3dac24589935402a53d0877d6"}
Jul 30 02:58:17.414089 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.413833    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj"
Jul 30 02:58:17.422363 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.422209    2565 scope.go:117] "RemoveContainer" containerID="9b4a29f3a2f8387c99478f9b758205365135bb81b07a79d94484c64def0ac9a3"
Jul 30 02:58:17.422569 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:58:17.422543    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b4a29f3a2f8387c99478f9b758205365135bb81b07a79d94484c64def0ac9a3\": container with ID starting with 9b4a29f3a2f8387c99478f9b758205365135bb81b07a79d94484c64def0ac9a3 not found: ID does not exist" containerID="9b4a29f3a2f8387c99478f9b758205365135bb81b07a79d94484c64def0ac9a3"
Jul 30 02:58:17.422680 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.422580    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b4a29f3a2f8387c99478f9b758205365135bb81b07a79d94484c64def0ac9a3"} err="failed to get container status \"9b4a29f3a2f8387c99478f9b758205365135bb81b07a79d94484c64def0ac9a3\": rpc error: code = NotFound desc = could not find container \"9b4a29f3a2f8387c99478f9b758205365135bb81b07a79d94484c64def0ac9a3\": container with ID starting with 9b4a29f3a2f8387c99478f9b758205365135bb81b07a79d94484c64def0ac9a3 not found: ID does not exist"
Jul 30 02:58:17.422680 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.422606    2565 scope.go:117] "RemoveContainer" containerID="fbcef63af7aebb4eae65825a336e9873f660f10633e4bf80153d843d687cd88f"
Jul 30 02:58:17.431508 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.431492    2565 scope.go:117] "RemoveContainer" containerID="fbcef63af7aebb4eae65825a336e9873f660f10633e4bf80153d843d687cd88f"
Jul 30 02:58:17.431728 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:58:17.431702    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbcef63af7aebb4eae65825a336e9873f660f10633e4bf80153d843d687cd88f\": container with ID starting with fbcef63af7aebb4eae65825a336e9873f660f10633e4bf80153d843d687cd88f not found: ID does not exist" containerID="fbcef63af7aebb4eae65825a336e9873f660f10633e4bf80153d843d687cd88f"
Jul 30 02:58:17.431799 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.431727    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbcef63af7aebb4eae65825a336e9873f660f10633e4bf80153d843d687cd88f"} err="failed to get container status \"fbcef63af7aebb4eae65825a336e9873f660f10633e4bf80153d843d687cd88f\": rpc error: code = NotFound desc = could not find container \"fbcef63af7aebb4eae65825a336e9873f660f10633e4bf80153d843d687cd88f\": container with ID starting with fbcef63af7aebb4eae65825a336e9873f660f10633e4bf80153d843d687cd88f not found: ID does not exist"
Jul 30 02:58:17.431799 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.431743    2565 scope.go:117] "RemoveContainer" containerID="ccaec3f6b4b21dfd3d142e5b0cffa4217cbc5b91b5fd381031ea45740d0a6d9a"
Jul 30 02:58:17.439918 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.439897    2565 scope.go:117] "RemoveContainer" containerID="ccaec3f6b4b21dfd3d142e5b0cffa4217cbc5b91b5fd381031ea45740d0a6d9a"
Jul 30 02:58:17.440209 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:58:17.440190    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccaec3f6b4b21dfd3d142e5b0cffa4217cbc5b91b5fd381031ea45740d0a6d9a\": container with ID starting with ccaec3f6b4b21dfd3d142e5b0cffa4217cbc5b91b5fd381031ea45740d0a6d9a not found: ID does not exist" containerID="ccaec3f6b4b21dfd3d142e5b0cffa4217cbc5b91b5fd381031ea45740d0a6d9a"
Jul 30 02:58:17.440261 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.440216    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccaec3f6b4b21dfd3d142e5b0cffa4217cbc5b91b5fd381031ea45740d0a6d9a"} err="failed to get container status \"ccaec3f6b4b21dfd3d142e5b0cffa4217cbc5b91b5fd381031ea45740d0a6d9a\": rpc error: code = NotFound desc = could not find container \"ccaec3f6b4b21dfd3d142e5b0cffa4217cbc5b91b5fd381031ea45740d0a6d9a\": container with ID starting with ccaec3f6b4b21dfd3d142e5b0cffa4217cbc5b91b5fd381031ea45740d0a6d9a not found: ID does not exist"
Jul 30 02:58:17.440389 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.440374    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t"]
Jul 30 02:58:17.444588 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.444567    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d054fc9a-6fd5d5659b-jc45t"]
Jul 30 02:58:17.456677 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.456655    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"]
Jul 30 02:58:17.463107 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.463085    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d054fc9a-db4fbddcb-qzbwn"]
Jul 30 02:58:17.475930 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.475909    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj"]
Jul 30 02:58:17.479807 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:17.479787    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d054fc9a-666df44c84-mhqkj"]
Jul 30 02:58:18.800762 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:18.800728    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5791e3fc-4f4c-4d1d-87dd-e33ba303d001" path="/var/lib/kubelet/pods/5791e3fc-4f4c-4d1d-87dd-e33ba303d001/volumes"
Jul 30 02:58:18.801229 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:18.801118    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="748363c8-9b89-459c-8f96-e226b8782854" path="/var/lib/kubelet/pods/748363c8-9b89-459c-8f96-e226b8782854/volumes"
Jul 30 02:58:18.801476 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:18.801462    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f117cabe-6077-47f1-ad63-8d0f18bd610f" path="/var/lib/kubelet/pods/f117cabe-6077-47f1-ad63-8d0f18bd610f/volumes"
Jul 30 02:58:21.136267 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:21.136232    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d054fc9a-zcq2w"]
Jul 30 02:58:21.140919 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:21.140892    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d054fc9a-zcq2w"]
Jul 30 02:58:21.279873 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:21.279840    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"]
Jul 30 02:58:21.280133 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:21.280105    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj" podUID="36a847ff-2a8a-42c3-9c88-302f2fded881" containerName="istio-proxy" containerID="cri-o://9e927b920d240855ea823cdce14e4c9fc9f007804d43d131a2a1e0926ed22958" gracePeriod=30
Jul 30 02:58:22.262630 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.262597    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"]
Jul 30 02:58:22.263002 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.262989    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f117cabe-6077-47f1-ad63-8d0f18bd610f" containerName="payload-pre-processing"
Jul 30 02:58:22.263002 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.263003    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f117cabe-6077-47f1-ad63-8d0f18bd610f" containerName="payload-pre-processing"
Jul 30 02:58:22.263076 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.263010    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="748363c8-9b89-459c-8f96-e226b8782854" containerName="maas-api"
Jul 30 02:58:22.263076 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.263016    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="748363c8-9b89-459c-8f96-e226b8782854" containerName="maas-api"
Jul 30 02:58:22.263076 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.263026    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5791e3fc-4f4c-4d1d-87dd-e33ba303d001" containerName="payload-processing"
Jul 30 02:58:22.263076 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.263031    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5791e3fc-4f4c-4d1d-87dd-e33ba303d001" containerName="payload-processing"
Jul 30 02:58:22.263076 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.263041    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="023ee300-8443-4cd8-aaee-4cb2ed21f6ff" containerName="revoke-keys"
Jul 30 02:58:22.263076 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.263046    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="023ee300-8443-4cd8-aaee-4cb2ed21f6ff" containerName="revoke-keys"
Jul 30 02:58:22.263288 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.263113    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="748363c8-9b89-459c-8f96-e226b8782854" containerName="maas-api"
Jul 30 02:58:22.263288 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.263123    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5791e3fc-4f4c-4d1d-87dd-e33ba303d001" containerName="payload-processing"
Jul 30 02:58:22.263288 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.263130    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="023ee300-8443-4cd8-aaee-4cb2ed21f6ff" containerName="revoke-keys"
Jul 30 02:58:22.263288 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.263136    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f117cabe-6077-47f1-ad63-8d0f18bd610f" containerName="payload-pre-processing"
Jul 30 02:58:22.268056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.268039    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.271114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.271094    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-9625a339-gw-openshift-default-dockercfg-2fns7\""
Jul 30 02:58:22.279801 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.279776    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"]
Jul 30 02:58:22.359630 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.359599    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-workload-certs\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.359782 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.359636    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-istio-envoy\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.359782 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.359658    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b21b996d-342e-4246-a08e-240064c3a814-istiod-ca-cert\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.359782 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.359706    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-istio-data\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.359782 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.359759    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-workload-socket\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.359782 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.359780    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b21b996d-342e-4246-a08e-240064c3a814-istio-token\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.359951 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.359807    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk7jx\" (UniqueName: \"kubernetes.io/projected/b21b996d-342e-4246-a08e-240064c3a814-kube-api-access-gk7jx\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.359951 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.359833    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-credential-socket\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.359951 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.359868    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b21b996d-342e-4246-a08e-240064c3a814-istio-podinfo\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.461190 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.461139    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-workload-certs\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.461190 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.461196    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-istio-envoy\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.461405 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.461301    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b21b996d-342e-4246-a08e-240064c3a814-istiod-ca-cert\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.461405 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.461338    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-istio-data\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.461519 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.461498    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-workload-socket\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.461581 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.461539    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b21b996d-342e-4246-a08e-240064c3a814-istio-token\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.461581 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.461508    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-workload-certs\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.461683 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.461577    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk7jx\" (UniqueName: \"kubernetes.io/projected/b21b996d-342e-4246-a08e-240064c3a814-kube-api-access-gk7jx\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.461683 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.461601    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-istio-data\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.461784 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.461720    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-credential-socket\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.461784 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.461730    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-workload-socket\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.461866 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.461804    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b21b996d-342e-4246-a08e-240064c3a814-istio-podinfo\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.462012 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.461990    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-credential-socket\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.462092 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.461995    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b21b996d-342e-4246-a08e-240064c3a814-istiod-ca-cert\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.463606 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.463582    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-istio-envoy\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.464160 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.464144    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b21b996d-342e-4246-a08e-240064c3a814-istio-podinfo\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.470525 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.470500    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b21b996d-342e-4246-a08e-240064c3a814-istio-token\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.470627 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.470549    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk7jx\" (UniqueName: \"kubernetes.io/projected/b21b996d-342e-4246-a08e-240064c3a814-kube-api-access-gk7jx\") pod \"e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") " pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.580764 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.580728    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:22.722620 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.722594    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"]
Jul 30 02:58:22.724520 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:58:22.724492    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21b996d_342e_4246_a08e_240064c3a814.slice/crio-450929800f7a28c354737549d216f30f8cd57a10cba9a121d05c50cc7515b723 WatchSource:0}: Error finding container 450929800f7a28c354737549d216f30f8cd57a10cba9a121d05c50cc7515b723: Status 404 returned error can't find the container with id 450929800f7a28c354737549d216f30f8cd57a10cba9a121d05c50cc7515b723
Jul 30 02:58:22.726799 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.726761    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:58:22.726862 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.726828    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:58:22.726862 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.726855    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:58:22.802538 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:22.802507    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="023ee300-8443-4cd8-aaee-4cb2ed21f6ff" path="/var/lib/kubelet/pods/023ee300-8443-4cd8-aaee-4cb2ed21f6ff/volumes"
Jul 30 02:58:23.452385 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:23.452352    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv" event={"ID":"b21b996d-342e-4246-a08e-240064c3a814","Type":"ContainerStarted","Data":"685643b7597199e1a8016d08d9b4e017488cd327dac0b3adb898f3c09c7a010b"}
Jul 30 02:58:23.452746 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:23.452403    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv" event={"ID":"b21b996d-342e-4246-a08e-240064c3a814","Type":"ContainerStarted","Data":"450929800f7a28c354737549d216f30f8cd57a10cba9a121d05c50cc7515b723"}
Jul 30 02:58:23.477595 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:23.477539    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv" podStartSLOduration=1.477519512 podStartE2EDuration="1.477519512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:58:23.472900855 +0000 UTC m=+1939.119684261" watchObservedRunningTime="2026-07-30 02:58:23.477519512 +0000 UTC m=+1939.124302918"
Jul 30 02:58:23.581427 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:23.581392    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:23.592252 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:23.592223    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:24.460673 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:24.460643    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:24.461579 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:24.461559    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:25.138488 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.138453    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"]
Jul 30 02:58:25.142475 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.142451    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"
Jul 30 02:58:25.145296 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.145274    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-9625a339-serving-cert\""
Jul 30 02:58:25.155097 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.155074    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"]
Jul 30 02:58:25.186719 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.186685    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xls9v\" (UniqueName: \"kubernetes.io/projected/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-kube-api-access-xls9v\") pod \"maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2\" (UID: \"92ac0132-7242-4fb7-9e3d-ecfbb98c04cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"
Jul 30 02:58:25.186719 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.186721    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-openshift-service-ca\") pod \"maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2\" (UID: \"92ac0132-7242-4fb7-9e3d-ecfbb98c04cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"
Jul 30 02:58:25.186911 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.186810    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-maas-api-tls\") pod \"maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2\" (UID: \"92ac0132-7242-4fb7-9e3d-ecfbb98c04cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"
Jul 30 02:58:25.287500 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.287467    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xls9v\" (UniqueName: \"kubernetes.io/projected/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-kube-api-access-xls9v\") pod \"maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2\" (UID: \"92ac0132-7242-4fb7-9e3d-ecfbb98c04cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"
Jul 30 02:58:25.287673 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.287511    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-openshift-service-ca\") pod \"maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2\" (UID: \"92ac0132-7242-4fb7-9e3d-ecfbb98c04cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"
Jul 30 02:58:25.287673 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.287592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-maas-api-tls\") pod \"maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2\" (UID: \"92ac0132-7242-4fb7-9e3d-ecfbb98c04cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"
Jul 30 02:58:25.287750 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:58:25.287692    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-serving-cert: secret "maas-api-e2e-migrate-9625a339-serving-cert" not found
Jul 30 02:58:25.287750 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:58:25.287747    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-maas-api-tls podName:92ac0132-7242-4fb7-9e3d-ecfbb98c04cf nodeName:}" failed. No retries permitted until 2026-07-30 02:58:25.787730824 +0000 UTC m=+1941.434514210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-maas-api-tls") pod "maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2" (UID: "92ac0132-7242-4fb7-9e3d-ecfbb98c04cf") : secret "maas-api-e2e-migrate-9625a339-serving-cert" not found
Jul 30 02:58:25.288926 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.288884    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-openshift-service-ca\") pod \"maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2\" (UID: \"92ac0132-7242-4fb7-9e3d-ecfbb98c04cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"
Jul 30 02:58:25.296957 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.296932    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xls9v\" (UniqueName: \"kubernetes.io/projected/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-kube-api-access-xls9v\") pod \"maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2\" (UID: \"92ac0132-7242-4fb7-9e3d-ecfbb98c04cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"
Jul 30 02:58:25.778739 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.778707    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp"]
Jul 30 02:58:25.784025 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.784004    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp"
Jul 30 02:58:25.787310 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.787287    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-9625a339\""
Jul 30 02:58:25.787417 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.787304    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-9625a339-dockercfg-qf2fq\""
Jul 30 02:58:25.792150 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.792111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-maas-api-tls\") pod \"maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2\" (UID: \"92ac0132-7242-4fb7-9e3d-ecfbb98c04cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"
Jul 30 02:58:25.795421 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.795398    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-maas-api-tls\") pod \"maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2\" (UID: \"92ac0132-7242-4fb7-9e3d-ecfbb98c04cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"
Jul 30 02:58:25.796562 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.796541    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp"]
Jul 30 02:58:25.893784 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.893723    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x76j\" (UniqueName: \"kubernetes.io/projected/ad2988ec-a744-49d4-a800-21a56534d890-kube-api-access-2x76j\") pod \"payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp\" (UID: \"ad2988ec-a744-49d4-a800-21a56534d890\") " pod="openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp"
Jul 30 02:58:25.894001 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.893910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2988ec-a744-49d4-a800-21a56534d890-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp\" (UID: \"ad2988ec-a744-49d4-a800-21a56534d890\") " pod="openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp"
Jul 30 02:58:25.994736 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.994705    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x76j\" (UniqueName: \"kubernetes.io/projected/ad2988ec-a744-49d4-a800-21a56534d890-kube-api-access-2x76j\") pod \"payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp\" (UID: \"ad2988ec-a744-49d4-a800-21a56534d890\") " pod="openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp"
Jul 30 02:58:25.994924 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.994813    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2988ec-a744-49d4-a800-21a56534d890-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp\" (UID: \"ad2988ec-a744-49d4-a800-21a56534d890\") " pod="openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp"
Jul 30 02:58:25.995483 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:25.995464    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2988ec-a744-49d4-a800-21a56534d890-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp\" (UID: \"ad2988ec-a744-49d4-a800-21a56534d890\") " pod="openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp"
Jul 30 02:58:26.003946 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.003918    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x76j\" (UniqueName: \"kubernetes.io/projected/ad2988ec-a744-49d4-a800-21a56534d890-kube-api-access-2x76j\") pod \"payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp\" (UID: \"ad2988ec-a744-49d4-a800-21a56534d890\") " pod="openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp"
Jul 30 02:58:26.054534 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.054506    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"
Jul 30 02:58:26.098812 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.098768    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp"
Jul 30 02:58:26.210464 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.210433    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"]
Jul 30 02:58:26.211870 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:58:26.211836    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ac0132_7242_4fb7_9e3d_ecfbb98c04cf.slice/crio-6bddf47f1c694ac15407b7a6e90606c8a2f07b7e0e2543163cb2dd4ea3a0c3b7 WatchSource:0}: Error finding container 6bddf47f1c694ac15407b7a6e90606c8a2f07b7e0e2543163cb2dd4ea3a0c3b7: Status 404 returned error can't find the container with id 6bddf47f1c694ac15407b7a6e90606c8a2f07b7e0e2543163cb2dd4ea3a0c3b7
Jul 30 02:58:26.259869 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.259843    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp"]
Jul 30 02:58:26.261184 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:58:26.261145    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2988ec_a744_49d4_a800_21a56534d890.slice/crio-5ac5495d39bc0ad61f23ace38f86b3ac8f642de5046265e30e823497cafb92ed WatchSource:0}: Error finding container 5ac5495d39bc0ad61f23ace38f86b3ac8f642de5046265e30e823497cafb92ed: Status 404 returned error can't find the container with id 5ac5495d39bc0ad61f23ace38f86b3ac8f642de5046265e30e823497cafb92ed
Jul 30 02:58:26.477098 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.476728    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2" event={"ID":"92ac0132-7242-4fb7-9e3d-ecfbb98c04cf","Type":"ContainerStarted","Data":"6bddf47f1c694ac15407b7a6e90606c8a2f07b7e0e2543163cb2dd4ea3a0c3b7"}
Jul 30 02:58:26.483286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.481341    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp" event={"ID":"ad2988ec-a744-49d4-a800-21a56534d890","Type":"ContainerStarted","Data":"67a3cbebec2f16c215f4b8c3742efc64bab0617c6fedf8bb348662c61e03aef5"}
Jul 30 02:58:26.483286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.481403    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp" event={"ID":"ad2988ec-a744-49d4-a800-21a56534d890","Type":"ContainerStarted","Data":"5ac5495d39bc0ad61f23ace38f86b3ac8f642de5046265e30e823497cafb92ed"}
Jul 30 02:58:26.483286 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.482342    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp"
Jul 30 02:58:26.487076 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.485624    2565 generic.go:356] "Generic (PLEG): container finished" podID="36a847ff-2a8a-42c3-9c88-302f2fded881" containerID="9e927b920d240855ea823cdce14e4c9fc9f007804d43d131a2a1e0926ed22958" exitCode=0
Jul 30 02:58:26.487076 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.485697    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj" event={"ID":"36a847ff-2a8a-42c3-9c88-302f2fded881","Type":"ContainerDied","Data":"9e927b920d240855ea823cdce14e4c9fc9f007804d43d131a2a1e0926ed22958"}
Jul 30 02:58:26.487076 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.485758    2565 scope.go:117] "RemoveContainer" containerID="658b7fe62e944eb1cff67a625dcb596d412d1baf635fde5c10eb909abae80d41"
Jul 30 02:58:26.500087 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.500022    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp" podStartSLOduration=1.5000011039999999 podStartE2EDuration="1.500001104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:58:26.499607402 +0000 UTC m=+1942.146390800" watchObservedRunningTime="2026-07-30 02:58:26.500001104 +0000 UTC m=+1942.146784510"
Jul 30 02:58:26.546459 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.546435    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:58:26.602114 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.602044    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-token\") pod \"36a847ff-2a8a-42c3-9c88-302f2fded881\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") "
Jul 30 02:58:26.602324 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.602134    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-podinfo\") pod \"36a847ff-2a8a-42c3-9c88-302f2fded881\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") "
Jul 30 02:58:26.602324 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.602211    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-credential-socket\") pod \"36a847ff-2a8a-42c3-9c88-302f2fded881\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") "
Jul 30 02:58:26.602324 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.602250    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a847ff-2a8a-42c3-9c88-302f2fded881-istiod-ca-cert\") pod \"36a847ff-2a8a-42c3-9c88-302f2fded881\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") "
Jul 30 02:58:26.602324 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.602296    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-envoy\") pod \"36a847ff-2a8a-42c3-9c88-302f2fded881\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") "
Jul 30 02:58:26.602324 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.602320    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-workload-socket\") pod \"36a847ff-2a8a-42c3-9c88-302f2fded881\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") "
Jul 30 02:58:26.602610 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.602351    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th5g9\" (UniqueName: \"kubernetes.io/projected/36a847ff-2a8a-42c3-9c88-302f2fded881-kube-api-access-th5g9\") pod \"36a847ff-2a8a-42c3-9c88-302f2fded881\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") "
Jul 30 02:58:26.602610 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.602379    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-workload-certs\") pod \"36a847ff-2a8a-42c3-9c88-302f2fded881\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") "
Jul 30 02:58:26.602610 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.602415    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-data\") pod \"36a847ff-2a8a-42c3-9c88-302f2fded881\" (UID: \"36a847ff-2a8a-42c3-9c88-302f2fded881\") "
Jul 30 02:58:26.602610 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.602501    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "36a847ff-2a8a-42c3-9c88-302f2fded881" (UID: "36a847ff-2a8a-42c3-9c88-302f2fded881"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:58:26.602818 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.602763    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-credential-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:58:26.602957 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.602929    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "36a847ff-2a8a-42c3-9c88-302f2fded881" (UID: "36a847ff-2a8a-42c3-9c88-302f2fded881"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:58:26.603027 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.602951    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "36a847ff-2a8a-42c3-9c88-302f2fded881" (UID: "36a847ff-2a8a-42c3-9c88-302f2fded881"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:58:26.603162 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.603140    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a847ff-2a8a-42c3-9c88-302f2fded881-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "36a847ff-2a8a-42c3-9c88-302f2fded881" (UID: "36a847ff-2a8a-42c3-9c88-302f2fded881"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 02:58:26.603544 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.603517    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-data" (OuterVolumeSpecName: "istio-data") pod "36a847ff-2a8a-42c3-9c88-302f2fded881" (UID: "36a847ff-2a8a-42c3-9c88-302f2fded881"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:58:26.605298 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.605085    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "36a847ff-2a8a-42c3-9c88-302f2fded881" (UID: "36a847ff-2a8a-42c3-9c88-302f2fded881"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:58:26.605413 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.605397    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-token" (OuterVolumeSpecName: "istio-token") pod "36a847ff-2a8a-42c3-9c88-302f2fded881" (UID: "36a847ff-2a8a-42c3-9c88-302f2fded881"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:58:26.605581 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.605557    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "36a847ff-2a8a-42c3-9c88-302f2fded881" (UID: "36a847ff-2a8a-42c3-9c88-302f2fded881"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 30 02:58:26.605696 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.605671    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a847ff-2a8a-42c3-9c88-302f2fded881-kube-api-access-th5g9" (OuterVolumeSpecName: "kube-api-access-th5g9") pod "36a847ff-2a8a-42c3-9c88-302f2fded881" (UID: "36a847ff-2a8a-42c3-9c88-302f2fded881"). InnerVolumeSpecName "kube-api-access-th5g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:58:26.703800 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.703763    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-token\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:58:26.703800 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.703799    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-podinfo\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:58:26.703994 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.703813    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a847ff-2a8a-42c3-9c88-302f2fded881-istiod-ca-cert\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:58:26.703994 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.703825    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-envoy\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:58:26.703994 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.703840    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-workload-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:58:26.703994 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.703852    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th5g9\" (UniqueName: \"kubernetes.io/projected/36a847ff-2a8a-42c3-9c88-302f2fded881-kube-api-access-th5g9\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:58:26.703994 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.703865    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-workload-certs\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:58:26.703994 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:26.703877    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a847ff-2a8a-42c3-9c88-302f2fded881-istio-data\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:58:27.494975 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:27.494945    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"
Jul 30 02:58:27.494975 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:27.494958    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj" event={"ID":"36a847ff-2a8a-42c3-9c88-302f2fded881","Type":"ContainerDied","Data":"7b3bb6e389cce8d8d997fedcd7b063bf940f77a0269b51d56e813635302e7d95"}
Jul 30 02:58:27.495522 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:27.495025    2565 scope.go:117] "RemoveContainer" containerID="9e927b920d240855ea823cdce14e4c9fc9f007804d43d131a2a1e0926ed22958"
Jul 30 02:58:27.497033 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:27.497005    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2" event={"ID":"92ac0132-7242-4fb7-9e3d-ecfbb98c04cf","Type":"ContainerStarted","Data":"0eeda726fb8c83176b1df8ef09d017c5331004fbbd903d94e219e2f7d109bcc6"}
Jul 30 02:58:27.497246 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:27.497216    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"
Jul 30 02:58:27.515327 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:27.515304    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"]
Jul 30 02:58:27.520632 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:27.520609    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d054fc9a-openshift-default-7f5f6f4996-vh7mj"]
Jul 30 02:58:27.533417 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:27.533359    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2" podStartSLOduration=2.142234736 podStartE2EDuration="2.533342576s" totalImagesPullingTime="391.10784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:58:27.532747591 +0000 UTC m=+1943.179530996" watchObservedRunningTime="2026-07-30 02:58:27.533342576 +0000 UTC m=+1943.180125982"
Jul 30 02:58:28.800377 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:28.800346    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36a847ff-2a8a-42c3-9c88-302f2fded881" path="/var/lib/kubelet/pods/36a847ff-2a8a-42c3-9c88-302f2fded881/volumes"
Jul 30 02:58:33.509960 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:33.509927    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"
Jul 30 02:58:38.506364 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:38.506331    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp"
Jul 30 02:58:40.502331 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:40.502295    2565 patch_prober.go:28] interesting pod/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 30 02:58:40.502745 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:40.502368    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv" podUID="b21b996d-342e-4246-a08e-240064c3a814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 30 02:58:41.591019 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:41.590985    2565 generic.go:356] "Generic (PLEG): container finished" podID="b21b996d-342e-4246-a08e-240064c3a814" containerID="685643b7597199e1a8016d08d9b4e017488cd327dac0b3adb898f3c09c7a010b" exitCode=0
Jul 30 02:58:41.591488 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:41.591030    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv" event={"ID":"b21b996d-342e-4246-a08e-240064c3a814","Type":"ContainerDied","Data":"685643b7597199e1a8016d08d9b4e017488cd327dac0b3adb898f3c09c7a010b"}
Jul 30 02:58:41.591488 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:41.591437    2565 scope.go:117] "RemoveContainer" containerID="685643b7597199e1a8016d08d9b4e017488cd327dac0b3adb898f3c09c7a010b"
Jul 30 02:58:41.592470 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:41.592443    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:58:41.592535 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:41.592509    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:58:41.592581 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:41.592541    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:58:42.601525 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:42.601479    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv" event={"ID":"b21b996d-342e-4246-a08e-240064c3a814","Type":"ContainerStarted","Data":"ff257bd7721ac70d663547ff52ff62fddaedb0a74d2a5538cec72098440cbbbd"}
Jul 30 02:58:43.581696 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:43.581654    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:43.581921 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:43.581789    2565 patch_prober.go:28] interesting pod/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 30 02:58:43.581921 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:43.581839    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv" podUID="b21b996d-342e-4246-a08e-240064c3a814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 30 02:58:44.583224 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:44.583179    2565 patch_prober.go:28] interesting pod/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 30 02:58:44.583747 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:44.583270    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv" podUID="b21b996d-342e-4246-a08e-240064c3a814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 30 02:58:45.581462 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:45.581419    2565 patch_prober.go:28] interesting pod/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 30 02:58:45.581656 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:45.581489    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv" podUID="b21b996d-342e-4246-a08e-240064c3a814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 30 02:58:46.585139 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:46.585107    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:46.585535 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:46.585395    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:46.586158 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:46.586137    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:58:49.893183 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:49.893128    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9625a339-2g8x9"]
Jul 30 02:58:49.893655 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:49.893545    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36a847ff-2a8a-42c3-9c88-302f2fded881" containerName="istio-proxy"
Jul 30 02:58:49.893655 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:49.893557    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="36a847ff-2a8a-42c3-9c88-302f2fded881" containerName="istio-proxy"
Jul 30 02:58:49.893655 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:49.893566    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36a847ff-2a8a-42c3-9c88-302f2fded881" containerName="istio-proxy"
Jul 30 02:58:49.893655 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:49.893571    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="36a847ff-2a8a-42c3-9c88-302f2fded881" containerName="istio-proxy"
Jul 30 02:58:49.893655 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:49.893638    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="36a847ff-2a8a-42c3-9c88-302f2fded881" containerName="istio-proxy"
Jul 30 02:58:49.893655 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:49.893647    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="36a847ff-2a8a-42c3-9c88-302f2fded881" containerName="istio-proxy"
Jul 30 02:58:49.896847 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:49.896831    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9625a339-2g8x9"
Jul 30 02:58:49.899700 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:49.899671    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2npk7\""
Jul 30 02:58:49.904107 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:49.904083    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9625a339-2g8x9"]
Jul 30 02:58:50.036765 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:50.036729    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/611682e8-dc05-4b48-9e05-bf3ff7c538f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9625a339-2g8x9\" (UID: \"611682e8-dc05-4b48-9e05-bf3ff7c538f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9625a339-2g8x9"
Jul 30 02:58:50.137512 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:50.137472    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/611682e8-dc05-4b48-9e05-bf3ff7c538f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9625a339-2g8x9\" (UID: \"611682e8-dc05-4b48-9e05-bf3ff7c538f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9625a339-2g8x9"
Jul 30 02:58:50.138087 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:50.138065    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/611682e8-dc05-4b48-9e05-bf3ff7c538f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9625a339-2g8x9\" (UID: \"611682e8-dc05-4b48-9e05-bf3ff7c538f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9625a339-2g8x9"
Jul 30 02:58:50.209661 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:50.209589    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9625a339-2g8x9"
Jul 30 02:58:50.344732 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:50.344707    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9625a339-2g8x9"]
Jul 30 02:58:50.346905 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:58:50.346863    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611682e8_dc05_4b48_9e05_bf3ff7c538f9.slice/crio-4ace8602e303555b1356b9331a0f2343ae6365ca22a1617947b5ddf5b4414623 WatchSource:0}: Error finding container 4ace8602e303555b1356b9331a0f2343ae6365ca22a1617947b5ddf5b4414623: Status 404 returned error can't find the container with id 4ace8602e303555b1356b9331a0f2343ae6365ca22a1617947b5ddf5b4414623
Jul 30 02:58:50.646094 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:50.646064    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9625a339-2g8x9" event={"ID":"611682e8-dc05-4b48-9e05-bf3ff7c538f9","Type":"ContainerStarted","Data":"4ace8602e303555b1356b9331a0f2343ae6365ca22a1617947b5ddf5b4414623"}
Jul 30 02:58:51.653218 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:51.653162    2565 generic.go:356] "Generic (PLEG): container finished" podID="611682e8-dc05-4b48-9e05-bf3ff7c538f9" containerID="595d3e6f553ea772f9d85459db70d112042134829d22e6a5f58b6ad68ec2912f" exitCode=0
Jul 30 02:58:51.653635 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:51.653267    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9625a339-2g8x9" event={"ID":"611682e8-dc05-4b48-9e05-bf3ff7c538f9","Type":"ContainerDied","Data":"595d3e6f553ea772f9d85459db70d112042134829d22e6a5f58b6ad68ec2912f"}
Jul 30 02:58:52.796051 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:52.796026    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9625a339-2g8x9"
Jul 30 02:58:52.963504 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:52.963419    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/611682e8-dc05-4b48-9e05-bf3ff7c538f9-maas-api-service-ca\") pod \"611682e8-dc05-4b48-9e05-bf3ff7c538f9\" (UID: \"611682e8-dc05-4b48-9e05-bf3ff7c538f9\") "
Jul 30 02:58:52.963838 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:52.963808    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/611682e8-dc05-4b48-9e05-bf3ff7c538f9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "611682e8-dc05-4b48-9e05-bf3ff7c538f9" (UID: "611682e8-dc05-4b48-9e05-bf3ff7c538f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 02:58:53.064634 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:53.064605    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/611682e8-dc05-4b48-9e05-bf3ff7c538f9-maas-api-service-ca\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:58:53.665822 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:53.665792    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9625a339-2g8x9"
Jul 30 02:58:53.666083 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:53.665838    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9625a339-2g8x9" event={"ID":"611682e8-dc05-4b48-9e05-bf3ff7c538f9","Type":"ContainerDied","Data":"4ace8602e303555b1356b9331a0f2343ae6365ca22a1617947b5ddf5b4414623"}
Jul 30 02:58:53.666083 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:58:53.665891    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ace8602e303555b1356b9331a0f2343ae6365ca22a1617947b5ddf5b4414623"
Jul 30 02:59:00.295737 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.295656    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"]
Jul 30 02:59:00.296215 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.295931    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2" podUID="92ac0132-7242-4fb7-9e3d-ecfbb98c04cf" containerName="maas-api" containerID="cri-o://0eeda726fb8c83176b1df8ef09d017c5331004fbbd903d94e219e2f7d109bcc6" gracePeriod=30
Jul 30 02:59:00.352198 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.351788    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp"]
Jul 30 02:59:00.352198 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.352052    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp" podUID="ad2988ec-a744-49d4-a800-21a56534d890" containerName="payload-processing" containerID="cri-o://67a3cbebec2f16c215f4b8c3742efc64bab0617c6fedf8bb348662c61e03aef5" gracePeriod=30
Jul 30 02:59:00.626437 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.626413    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"
Jul 30 02:59:00.661937 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.661911    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp"
Jul 30 02:59:00.707144 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.707095    2565 generic.go:356] "Generic (PLEG): container finished" podID="92ac0132-7242-4fb7-9e3d-ecfbb98c04cf" containerID="0eeda726fb8c83176b1df8ef09d017c5331004fbbd903d94e219e2f7d109bcc6" exitCode=0
Jul 30 02:59:00.707374 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.707147    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"
Jul 30 02:59:00.707665 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.707215    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2" event={"ID":"92ac0132-7242-4fb7-9e3d-ecfbb98c04cf","Type":"ContainerDied","Data":"0eeda726fb8c83176b1df8ef09d017c5331004fbbd903d94e219e2f7d109bcc6"}
Jul 30 02:59:00.707804 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.707784    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2" event={"ID":"92ac0132-7242-4fb7-9e3d-ecfbb98c04cf","Type":"ContainerDied","Data":"6bddf47f1c694ac15407b7a6e90606c8a2f07b7e0e2543163cb2dd4ea3a0c3b7"}
Jul 30 02:59:00.707868 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.707824    2565 scope.go:117] "RemoveContainer" containerID="0eeda726fb8c83176b1df8ef09d017c5331004fbbd903d94e219e2f7d109bcc6"
Jul 30 02:59:00.709322 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.709220    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad2988ec-a744-49d4-a800-21a56534d890" containerID="67a3cbebec2f16c215f4b8c3742efc64bab0617c6fedf8bb348662c61e03aef5" exitCode=0
Jul 30 02:59:00.709322 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.709297    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp" event={"ID":"ad2988ec-a744-49d4-a800-21a56534d890","Type":"ContainerDied","Data":"67a3cbebec2f16c215f4b8c3742efc64bab0617c6fedf8bb348662c61e03aef5"}
Jul 30 02:59:00.709454 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.709330    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp"
Jul 30 02:59:00.709454 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.709342    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp" event={"ID":"ad2988ec-a744-49d4-a800-21a56534d890","Type":"ContainerDied","Data":"5ac5495d39bc0ad61f23ace38f86b3ac8f642de5046265e30e823497cafb92ed"}
Jul 30 02:59:00.720800 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.720779    2565 scope.go:117] "RemoveContainer" containerID="0eeda726fb8c83176b1df8ef09d017c5331004fbbd903d94e219e2f7d109bcc6"
Jul 30 02:59:00.721056 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:59:00.721036    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eeda726fb8c83176b1df8ef09d017c5331004fbbd903d94e219e2f7d109bcc6\": container with ID starting with 0eeda726fb8c83176b1df8ef09d017c5331004fbbd903d94e219e2f7d109bcc6 not found: ID does not exist" containerID="0eeda726fb8c83176b1df8ef09d017c5331004fbbd903d94e219e2f7d109bcc6"
Jul 30 02:59:00.721154 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.721063    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eeda726fb8c83176b1df8ef09d017c5331004fbbd903d94e219e2f7d109bcc6"} err="failed to get container status \"0eeda726fb8c83176b1df8ef09d017c5331004fbbd903d94e219e2f7d109bcc6\": rpc error: code = NotFound desc = could not find container \"0eeda726fb8c83176b1df8ef09d017c5331004fbbd903d94e219e2f7d109bcc6\": container with ID starting with 0eeda726fb8c83176b1df8ef09d017c5331004fbbd903d94e219e2f7d109bcc6 not found: ID does not exist"
Jul 30 02:59:00.721154 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.721086    2565 scope.go:117] "RemoveContainer" containerID="67a3cbebec2f16c215f4b8c3742efc64bab0617c6fedf8bb348662c61e03aef5"
Jul 30 02:59:00.730302 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.730287    2565 scope.go:117] "RemoveContainer" containerID="67a3cbebec2f16c215f4b8c3742efc64bab0617c6fedf8bb348662c61e03aef5"
Jul 30 02:59:00.730539 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:59:00.730516    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a3cbebec2f16c215f4b8c3742efc64bab0617c6fedf8bb348662c61e03aef5\": container with ID starting with 67a3cbebec2f16c215f4b8c3742efc64bab0617c6fedf8bb348662c61e03aef5 not found: ID does not exist" containerID="67a3cbebec2f16c215f4b8c3742efc64bab0617c6fedf8bb348662c61e03aef5"
Jul 30 02:59:00.730594 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.730545    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a3cbebec2f16c215f4b8c3742efc64bab0617c6fedf8bb348662c61e03aef5"} err="failed to get container status \"67a3cbebec2f16c215f4b8c3742efc64bab0617c6fedf8bb348662c61e03aef5\": rpc error: code = NotFound desc = could not find container \"67a3cbebec2f16c215f4b8c3742efc64bab0617c6fedf8bb348662c61e03aef5\": container with ID starting with 67a3cbebec2f16c215f4b8c3742efc64bab0617c6fedf8bb348662c61e03aef5 not found: ID does not exist"
Jul 30 02:59:00.740325 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.740301    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-maas-api-tls\") pod \"92ac0132-7242-4fb7-9e3d-ecfbb98c04cf\" (UID: \"92ac0132-7242-4fb7-9e3d-ecfbb98c04cf\") "
Jul 30 02:59:00.740453 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.740437    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-openshift-service-ca\") pod \"92ac0132-7242-4fb7-9e3d-ecfbb98c04cf\" (UID: \"92ac0132-7242-4fb7-9e3d-ecfbb98c04cf\") "
Jul 30 02:59:00.740517 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.740504    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xls9v\" (UniqueName: \"kubernetes.io/projected/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-kube-api-access-xls9v\") pod \"92ac0132-7242-4fb7-9e3d-ecfbb98c04cf\" (UID: \"92ac0132-7242-4fb7-9e3d-ecfbb98c04cf\") "
Jul 30 02:59:00.740763 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.740736    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "92ac0132-7242-4fb7-9e3d-ecfbb98c04cf" (UID: "92ac0132-7242-4fb7-9e3d-ecfbb98c04cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 02:59:00.742386 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.742362    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "92ac0132-7242-4fb7-9e3d-ecfbb98c04cf" (UID: "92ac0132-7242-4fb7-9e3d-ecfbb98c04cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 02:59:00.742560 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.742542    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-kube-api-access-xls9v" (OuterVolumeSpecName: "kube-api-access-xls9v") pod "92ac0132-7242-4fb7-9e3d-ecfbb98c04cf" (UID: "92ac0132-7242-4fb7-9e3d-ecfbb98c04cf"). InnerVolumeSpecName "kube-api-access-xls9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:59:00.841820 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.841748    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2x76j\" (UniqueName: \"kubernetes.io/projected/ad2988ec-a744-49d4-a800-21a56534d890-kube-api-access-2x76j\") pod \"ad2988ec-a744-49d4-a800-21a56534d890\" (UID: \"ad2988ec-a744-49d4-a800-21a56534d890\") "
Jul 30 02:59:00.841820 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.841780    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2988ec-a744-49d4-a800-21a56534d890-plugins-config-volume\") pod \"ad2988ec-a744-49d4-a800-21a56534d890\" (UID: \"ad2988ec-a744-49d4-a800-21a56534d890\") "
Jul 30 02:59:00.841998 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.841940    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-openshift-service-ca\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:00.841998 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.841952    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xls9v\" (UniqueName: \"kubernetes.io/projected/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-kube-api-access-xls9v\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:00.841998 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.841963    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf-maas-api-tls\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:00.842231 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.842209    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2988ec-a744-49d4-a800-21a56534d890-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad2988ec-a744-49d4-a800-21a56534d890" (UID: "ad2988ec-a744-49d4-a800-21a56534d890"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 02:59:00.843932 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.843911    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2988ec-a744-49d4-a800-21a56534d890-kube-api-access-2x76j" (OuterVolumeSpecName: "kube-api-access-2x76j") pod "ad2988ec-a744-49d4-a800-21a56534d890" (UID: "ad2988ec-a744-49d4-a800-21a56534d890"). InnerVolumeSpecName "kube-api-access-2x76j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:59:00.943240 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.943211    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x76j\" (UniqueName: \"kubernetes.io/projected/ad2988ec-a744-49d4-a800-21a56534d890-kube-api-access-2x76j\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:00.943240 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:00.943236    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2988ec-a744-49d4-a800-21a56534d890-plugins-config-volume\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:01.023267 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:01.023239    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"]
Jul 30 02:59:01.028765 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:01.028738    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9625a339-9cf4d8949-25wv2"]
Jul 30 02:59:01.039572 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:01.039545    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp"]
Jul 30 02:59:01.044985 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:01.044959    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9625a339-695c8d557c-4nhsp"]
Jul 30 02:59:02.800756 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:02.800723    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92ac0132-7242-4fb7-9e3d-ecfbb98c04cf" path="/var/lib/kubelet/pods/92ac0132-7242-4fb7-9e3d-ecfbb98c04cf/volumes"
Jul 30 02:59:02.801153 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:02.801090    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad2988ec-a744-49d4-a800-21a56534d890" path="/var/lib/kubelet/pods/ad2988ec-a744-49d4-a800-21a56534d890/volumes"
Jul 30 02:59:04.977226 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:04.977197    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9625a339-2g8x9"]
Jul 30 02:59:04.979758 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:04.979733    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9625a339-2g8x9"]
Jul 30 02:59:05.110532 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.110499    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"]
Jul 30 02:59:05.110824 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.110761    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv" podUID="b21b996d-342e-4246-a08e-240064c3a814" containerName="istio-proxy" containerID="cri-o://ff257bd7721ac70d663547ff52ff62fddaedb0a74d2a5538cec72098440cbbbd" gracePeriod=30
Jul 30 02:59:05.882497 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.882467    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"]
Jul 30 02:59:05.882873 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.882859    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92ac0132-7242-4fb7-9e3d-ecfbb98c04cf" containerName="maas-api"
Jul 30 02:59:05.882924 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.882874    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="92ac0132-7242-4fb7-9e3d-ecfbb98c04cf" containerName="maas-api"
Jul 30 02:59:05.882924 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.882886    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="611682e8-dc05-4b48-9e05-bf3ff7c538f9" containerName="revoke-keys"
Jul 30 02:59:05.882924 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.882891    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="611682e8-dc05-4b48-9e05-bf3ff7c538f9" containerName="revoke-keys"
Jul 30 02:59:05.882924 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.882899    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad2988ec-a744-49d4-a800-21a56534d890" containerName="payload-processing"
Jul 30 02:59:05.882924 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.882905    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad2988ec-a744-49d4-a800-21a56534d890" containerName="payload-processing"
Jul 30 02:59:05.883099 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.882962    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="611682e8-dc05-4b48-9e05-bf3ff7c538f9" containerName="revoke-keys"
Jul 30 02:59:05.883099 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.882972    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad2988ec-a744-49d4-a800-21a56534d890" containerName="payload-processing"
Jul 30 02:59:05.883099 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.882981    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="92ac0132-7242-4fb7-9e3d-ecfbb98c04cf" containerName="maas-api"
Jul 30 02:59:05.887835 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.887815    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:05.890560 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.890541    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4fcfd02b-openshift-default-dockercfg-lfbqz\""
Jul 30 02:59:05.898015 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.897993    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"]
Jul 30 02:59:05.993177 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.993044    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-credential-socket\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:05.993177 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.993116    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e866047e-a275-4432-9476-19f88302a32c-istiod-ca-cert\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:05.993650 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.993224    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e866047e-a275-4432-9476-19f88302a32c-istio-podinfo\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:05.993650 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.993283    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-istio-envoy\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:05.993650 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.993319    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-istio-data\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:05.993650 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.993344    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e866047e-a275-4432-9476-19f88302a32c-istio-token\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:05.993650 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.993372    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-workload-certs\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:05.993650 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.993451    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-workload-socket\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:05.993650 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:05.993520    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pf47\" (UniqueName: \"kubernetes.io/projected/e866047e-a275-4432-9476-19f88302a32c-kube-api-access-2pf47\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:06.094970 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.094934    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-istio-envoy\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:06.094970 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.094974    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-istio-data\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:06.095278 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.094992    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e866047e-a275-4432-9476-19f88302a32c-istio-token\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:06.095278 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.095132    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-workload-certs\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:06.095278 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.095220    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-workload-socket\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:06.095465 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.095279    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pf47\" (UniqueName: \"kubernetes.io/projected/e866047e-a275-4432-9476-19f88302a32c-kube-api-access-2pf47\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:06.095465 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.095358    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-credential-socket\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:06.095465 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.095413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e866047e-a275-4432-9476-19f88302a32c-istiod-ca-cert\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:06.095465 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.095447    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e866047e-a275-4432-9476-19f88302a32c-istio-podinfo\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:06.095656 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.095472    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-istio-data\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:06.095656 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.095637    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-workload-certs\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:06.095756 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.095694    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-workload-socket\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:06.095756 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.095712    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-credential-socket\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:06.096223 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.096189    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e866047e-a275-4432-9476-19f88302a32c-istiod-ca-cert\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:06.098053 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.098011    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-istio-envoy\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:06.098934 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.098913    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e866047e-a275-4432-9476-19f88302a32c-istio-podinfo\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:06.102909 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.102887    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pf47\" (UniqueName: \"kubernetes.io/projected/e866047e-a275-4432-9476-19f88302a32c-kube-api-access-2pf47\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:06.103030 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.102973    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e866047e-a275-4432-9476-19f88302a32c-istio-token\") pod \"e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") " pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:06.200211 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.200097    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:06.341258 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.341228    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"]
Jul 30 02:59:06.342555 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:59:06.342525    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode866047e_a275_4432_9476_19f88302a32c.slice/crio-74bc4fb7817b7055ccd7674442b659f4b863a26bca216c1a65dbcdd0b0db9acf WatchSource:0}: Error finding container 74bc4fb7817b7055ccd7674442b659f4b863a26bca216c1a65dbcdd0b0db9acf: Status 404 returned error can't find the container with id 74bc4fb7817b7055ccd7674442b659f4b863a26bca216c1a65dbcdd0b0db9acf
Jul 30 02:59:06.344943 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.344913    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:59:06.345013 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.344983    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:59:06.345051 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.345013    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:59:06.747970 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.747934    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9" event={"ID":"e866047e-a275-4432-9476-19f88302a32c","Type":"ContainerStarted","Data":"fd70eae7f0bc518cf9e61e7feee7191078161b9a4209a5674e9453fe63dc6ccf"}
Jul 30 02:59:06.748147 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.747990    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9" event={"ID":"e866047e-a275-4432-9476-19f88302a32c","Type":"ContainerStarted","Data":"74bc4fb7817b7055ccd7674442b659f4b863a26bca216c1a65dbcdd0b0db9acf"}
Jul 30 02:59:06.768181 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.768120    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9" podStartSLOduration=1.768106073 podStartE2EDuration="1.768106073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:59:06.767037915 +0000 UTC m=+1982.413821315" watchObservedRunningTime="2026-07-30 02:59:06.768106073 +0000 UTC m=+1982.414889478"
Jul 30 02:59:06.803269 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:06.803228    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="611682e8-dc05-4b48-9e05-bf3ff7c538f9" path="/var/lib/kubelet/pods/611682e8-dc05-4b48-9e05-bf3ff7c538f9/volumes"
Jul 30 02:59:07.200346 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:07.200299    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:07.205266 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:07.205245    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:07.753143 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:07.753111    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:07.754121 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:07.754100    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:08.331877 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:08.329021    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"]
Jul 30 02:59:08.334138 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:08.334110    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"
Jul 30 02:59:08.338623 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:08.338560    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4fcfd02b-serving-cert\""
Jul 30 02:59:08.341443 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:08.341416    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"]
Jul 30 02:59:08.417276 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:08.417235    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djhqz\" (UniqueName: \"kubernetes.io/projected/e7150223-1c12-4051-925e-533b69a63e1a-kube-api-access-djhqz\") pod \"maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm\" (UID: \"e7150223-1c12-4051-925e-533b69a63e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"
Jul 30 02:59:08.417276 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:08.417272    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7150223-1c12-4051-925e-533b69a63e1a-openshift-service-ca\") pod \"maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm\" (UID: \"e7150223-1c12-4051-925e-533b69a63e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"
Jul 30 02:59:08.417539 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:08.417300    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7150223-1c12-4051-925e-533b69a63e1a-maas-api-tls\") pod \"maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm\" (UID: \"e7150223-1c12-4051-925e-533b69a63e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"
Jul 30 02:59:08.518479 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:08.518442    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhqz\" (UniqueName: \"kubernetes.io/projected/e7150223-1c12-4051-925e-533b69a63e1a-kube-api-access-djhqz\") pod \"maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm\" (UID: \"e7150223-1c12-4051-925e-533b69a63e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"
Jul 30 02:59:08.518674 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:08.518487    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7150223-1c12-4051-925e-533b69a63e1a-openshift-service-ca\") pod \"maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm\" (UID: \"e7150223-1c12-4051-925e-533b69a63e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"
Jul 30 02:59:08.518674 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:08.518527    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7150223-1c12-4051-925e-533b69a63e1a-maas-api-tls\") pod \"maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm\" (UID: \"e7150223-1c12-4051-925e-533b69a63e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"
Jul 30 02:59:08.519193 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:08.519129    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7150223-1c12-4051-925e-533b69a63e1a-openshift-service-ca\") pod \"maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm\" (UID: \"e7150223-1c12-4051-925e-533b69a63e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"
Jul 30 02:59:08.521094 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:08.521062    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7150223-1c12-4051-925e-533b69a63e1a-maas-api-tls\") pod \"maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm\" (UID: \"e7150223-1c12-4051-925e-533b69a63e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"
Jul 30 02:59:08.527041 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:08.527019    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djhqz\" (UniqueName: \"kubernetes.io/projected/e7150223-1c12-4051-925e-533b69a63e1a-kube-api-access-djhqz\") pod \"maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm\" (UID: \"e7150223-1c12-4051-925e-533b69a63e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"
Jul 30 02:59:08.647503 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:08.647427    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"
Jul 30 02:59:08.809552 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:08.809529    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"]
Jul 30 02:59:08.811289 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:59:08.811240    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7150223_1c12_4051_925e_533b69a63e1a.slice/crio-450997f88e3c5af89852c9fa11b1206000e3444f65285948aa2576cf71ec9df0 WatchSource:0}: Error finding container 450997f88e3c5af89852c9fa11b1206000e3444f65285948aa2576cf71ec9df0: Status 404 returned error can't find the container with id 450997f88e3c5af89852c9fa11b1206000e3444f65285948aa2576cf71ec9df0
Jul 30 02:59:09.765990 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:09.765903    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm" event={"ID":"e7150223-1c12-4051-925e-533b69a63e1a","Type":"ContainerStarted","Data":"2c3bbb10fc113978206f8edb9d0587d126f98410d460d99ead89c3a81a265b5b"}
Jul 30 02:59:09.765990 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:09.765956    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm" event={"ID":"e7150223-1c12-4051-925e-533b69a63e1a","Type":"ContainerStarted","Data":"450997f88e3c5af89852c9fa11b1206000e3444f65285948aa2576cf71ec9df0"}
Jul 30 02:59:09.766496 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:09.766008    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"
Jul 30 02:59:09.783313 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:09.783260    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm" podStartSLOduration=1.306484017 podStartE2EDuration="1.783242797s" totalImagesPullingTime="476.75878ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:59:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:59:09.782237059 +0000 UTC m=+1985.429020465" watchObservedRunningTime="2026-07-30 02:59:09.783242797 +0000 UTC m=+1985.430026229"
Jul 30 02:59:10.380665 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.380603    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:59:10.437253 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.437224    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-istio-data\") pod \"b21b996d-342e-4246-a08e-240064c3a814\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") "
Jul 30 02:59:10.437253 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.437255    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-credential-socket\") pod \"b21b996d-342e-4246-a08e-240064c3a814\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") "
Jul 30 02:59:10.437481 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.437286    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk7jx\" (UniqueName: \"kubernetes.io/projected/b21b996d-342e-4246-a08e-240064c3a814-kube-api-access-gk7jx\") pod \"b21b996d-342e-4246-a08e-240064c3a814\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") "
Jul 30 02:59:10.437481 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.437311    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b21b996d-342e-4246-a08e-240064c3a814-istio-podinfo\") pod \"b21b996d-342e-4246-a08e-240064c3a814\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") "
Jul 30 02:59:10.437481 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.437372    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b21b996d-342e-4246-a08e-240064c3a814-istiod-ca-cert\") pod \"b21b996d-342e-4246-a08e-240064c3a814\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") "
Jul 30 02:59:10.437481 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.437387    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b21b996d-342e-4246-a08e-240064c3a814-istio-token\") pod \"b21b996d-342e-4246-a08e-240064c3a814\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") "
Jul 30 02:59:10.437481 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.437413    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-workload-certs\") pod \"b21b996d-342e-4246-a08e-240064c3a814\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") "
Jul 30 02:59:10.437481 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.437443    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-workload-socket\") pod \"b21b996d-342e-4246-a08e-240064c3a814\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") "
Jul 30 02:59:10.437481 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.437479    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-istio-envoy\") pod \"b21b996d-342e-4246-a08e-240064c3a814\" (UID: \"b21b996d-342e-4246-a08e-240064c3a814\") "
Jul 30 02:59:10.437827 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.437524    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b21b996d-342e-4246-a08e-240064c3a814" (UID: "b21b996d-342e-4246-a08e-240064c3a814"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:59:10.437827 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.437766    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b21b996d-342e-4246-a08e-240064c3a814" (UID: "b21b996d-342e-4246-a08e-240064c3a814"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:59:10.437827 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.437815    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b21b996d-342e-4246-a08e-240064c3a814" (UID: "b21b996d-342e-4246-a08e-240064c3a814"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:59:10.437827 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.437822    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-credential-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:10.437985 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.437860    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b21b996d-342e-4246-a08e-240064c3a814-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b21b996d-342e-4246-a08e-240064c3a814" (UID: "b21b996d-342e-4246-a08e-240064c3a814"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 02:59:10.438397 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.438368    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-istio-data" (OuterVolumeSpecName: "istio-data") pod "b21b996d-342e-4246-a08e-240064c3a814" (UID: "b21b996d-342e-4246-a08e-240064c3a814"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:59:10.439801 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.439769    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b21b996d-342e-4246-a08e-240064c3a814-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b21b996d-342e-4246-a08e-240064c3a814" (UID: "b21b996d-342e-4246-a08e-240064c3a814"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 30 02:59:10.439926 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.439828    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21b996d-342e-4246-a08e-240064c3a814-kube-api-access-gk7jx" (OuterVolumeSpecName: "kube-api-access-gk7jx") pod "b21b996d-342e-4246-a08e-240064c3a814" (UID: "b21b996d-342e-4246-a08e-240064c3a814"). InnerVolumeSpecName "kube-api-access-gk7jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:59:10.439926 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.439845    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21b996d-342e-4246-a08e-240064c3a814-istio-token" (OuterVolumeSpecName: "istio-token") pod "b21b996d-342e-4246-a08e-240064c3a814" (UID: "b21b996d-342e-4246-a08e-240064c3a814"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:59:10.439926 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.439854    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b21b996d-342e-4246-a08e-240064c3a814" (UID: "b21b996d-342e-4246-a08e-240064c3a814"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:59:10.538440 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.538411    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-istio-data\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:10.538440 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.538439    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk7jx\" (UniqueName: \"kubernetes.io/projected/b21b996d-342e-4246-a08e-240064c3a814-kube-api-access-gk7jx\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:10.538628 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.538449    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b21b996d-342e-4246-a08e-240064c3a814-istio-podinfo\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:10.538628 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.538457    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b21b996d-342e-4246-a08e-240064c3a814-istiod-ca-cert\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:10.538628 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.538465    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b21b996d-342e-4246-a08e-240064c3a814-istio-token\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:10.538628 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.538475    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-workload-certs\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:10.538628 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.538483    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-workload-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:10.538628 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.538491    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b21b996d-342e-4246-a08e-240064c3a814-istio-envoy\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:10.773133 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.773030    2565 generic.go:356] "Generic (PLEG): container finished" podID="b21b996d-342e-4246-a08e-240064c3a814" containerID="ff257bd7721ac70d663547ff52ff62fddaedb0a74d2a5538cec72098440cbbbd" exitCode=0
Jul 30 02:59:10.773133 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.773098    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"
Jul 30 02:59:10.773133 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.773100    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv" event={"ID":"b21b996d-342e-4246-a08e-240064c3a814","Type":"ContainerDied","Data":"ff257bd7721ac70d663547ff52ff62fddaedb0a74d2a5538cec72098440cbbbd"}
Jul 30 02:59:10.773697 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.773160    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv" event={"ID":"b21b996d-342e-4246-a08e-240064c3a814","Type":"ContainerDied","Data":"450929800f7a28c354737549d216f30f8cd57a10cba9a121d05c50cc7515b723"}
Jul 30 02:59:10.773697 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.773203    2565 scope.go:117] "RemoveContainer" containerID="ff257bd7721ac70d663547ff52ff62fddaedb0a74d2a5538cec72098440cbbbd"
Jul 30 02:59:10.784455 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.784437    2565 scope.go:117] "RemoveContainer" containerID="685643b7597199e1a8016d08d9b4e017488cd327dac0b3adb898f3c09c7a010b"
Jul 30 02:59:10.795041 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.795016    2565 scope.go:117] "RemoveContainer" containerID="ff257bd7721ac70d663547ff52ff62fddaedb0a74d2a5538cec72098440cbbbd"
Jul 30 02:59:10.795438 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:59:10.795411    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff257bd7721ac70d663547ff52ff62fddaedb0a74d2a5538cec72098440cbbbd\": container with ID starting with ff257bd7721ac70d663547ff52ff62fddaedb0a74d2a5538cec72098440cbbbd not found: ID does not exist" containerID="ff257bd7721ac70d663547ff52ff62fddaedb0a74d2a5538cec72098440cbbbd"
Jul 30 02:59:10.795520 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.795452    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff257bd7721ac70d663547ff52ff62fddaedb0a74d2a5538cec72098440cbbbd"} err="failed to get container status \"ff257bd7721ac70d663547ff52ff62fddaedb0a74d2a5538cec72098440cbbbd\": rpc error: code = NotFound desc = could not find container \"ff257bd7721ac70d663547ff52ff62fddaedb0a74d2a5538cec72098440cbbbd\": container with ID starting with ff257bd7721ac70d663547ff52ff62fddaedb0a74d2a5538cec72098440cbbbd not found: ID does not exist"
Jul 30 02:59:10.795520 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.795477    2565 scope.go:117] "RemoveContainer" containerID="685643b7597199e1a8016d08d9b4e017488cd327dac0b3adb898f3c09c7a010b"
Jul 30 02:59:10.795781 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:59:10.795760    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685643b7597199e1a8016d08d9b4e017488cd327dac0b3adb898f3c09c7a010b\": container with ID starting with 685643b7597199e1a8016d08d9b4e017488cd327dac0b3adb898f3c09c7a010b not found: ID does not exist" containerID="685643b7597199e1a8016d08d9b4e017488cd327dac0b3adb898f3c09c7a010b"
Jul 30 02:59:10.795829 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.795791    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685643b7597199e1a8016d08d9b4e017488cd327dac0b3adb898f3c09c7a010b"} err="failed to get container status \"685643b7597199e1a8016d08d9b4e017488cd327dac0b3adb898f3c09c7a010b\": rpc error: code = NotFound desc = could not find container \"685643b7597199e1a8016d08d9b4e017488cd327dac0b3adb898f3c09c7a010b\": container with ID starting with 685643b7597199e1a8016d08d9b4e017488cd327dac0b3adb898f3c09c7a010b not found: ID does not exist"
Jul 30 02:59:10.800784 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.800762    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"]
Jul 30 02:59:10.805529 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:10.805507    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-9625a339-gw-openshift-default-7c944fd86d-mj6rv"]
Jul 30 02:59:12.800834 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:12.800795    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b21b996d-342e-4246-a08e-240064c3a814" path="/var/lib/kubelet/pods/b21b996d-342e-4246-a08e-240064c3a814/volumes"
Jul 30 02:59:15.779099 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:15.779072    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"
Jul 30 02:59:18.484820 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:18.484774    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4fcfd02b-kx2ff"]
Jul 30 02:59:18.485471 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:18.485443    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b21b996d-342e-4246-a08e-240064c3a814" containerName="istio-proxy"
Jul 30 02:59:18.485559 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:18.485474    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b21b996d-342e-4246-a08e-240064c3a814" containerName="istio-proxy"
Jul 30 02:59:18.485559 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:18.485501    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b21b996d-342e-4246-a08e-240064c3a814" containerName="istio-proxy"
Jul 30 02:59:18.485559 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:18.485511    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b21b996d-342e-4246-a08e-240064c3a814" containerName="istio-proxy"
Jul 30 02:59:18.485725 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:18.485618    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b21b996d-342e-4246-a08e-240064c3a814" containerName="istio-proxy"
Jul 30 02:59:18.485725 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:18.485632    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b21b996d-342e-4246-a08e-240064c3a814" containerName="istio-proxy"
Jul 30 02:59:18.490716 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:18.490685    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4fcfd02b-kx2ff"
Jul 30 02:59:18.494736 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:18.494707    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2npk7\""
Jul 30 02:59:18.500385 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:18.500356    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4fcfd02b-kx2ff"]
Jul 30 02:59:18.619636 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:18.619546    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58ae441-7361-4999-a8dc-ee5f8b2b9d57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4fcfd02b-kx2ff\" (UID: \"a58ae441-7361-4999-a8dc-ee5f8b2b9d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4fcfd02b-kx2ff"
Jul 30 02:59:18.720771 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:18.720731    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58ae441-7361-4999-a8dc-ee5f8b2b9d57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4fcfd02b-kx2ff\" (UID: \"a58ae441-7361-4999-a8dc-ee5f8b2b9d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4fcfd02b-kx2ff"
Jul 30 02:59:18.721472 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:18.721444    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58ae441-7361-4999-a8dc-ee5f8b2b9d57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4fcfd02b-kx2ff\" (UID: \"a58ae441-7361-4999-a8dc-ee5f8b2b9d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4fcfd02b-kx2ff"
Jul 30 02:59:18.806470 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:18.806435    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4fcfd02b-kx2ff"
Jul 30 02:59:18.967000 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:18.966970    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4fcfd02b-kx2ff"]
Jul 30 02:59:18.968979 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:59:18.968944    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58ae441_7361_4999_a8dc_ee5f8b2b9d57.slice/crio-8533b065a1989a26ecf2cf881e8032bb1a045e4000b74396336045d7aba086aa WatchSource:0}: Error finding container 8533b065a1989a26ecf2cf881e8032bb1a045e4000b74396336045d7aba086aa: Status 404 returned error can't find the container with id 8533b065a1989a26ecf2cf881e8032bb1a045e4000b74396336045d7aba086aa
Jul 30 02:59:19.839016 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:19.838983    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4fcfd02b-kx2ff" event={"ID":"a58ae441-7361-4999-a8dc-ee5f8b2b9d57","Type":"ContainerStarted","Data":"4d8443a1a18ffe3142a4180221bef139e3d21d675a773df7971e0b474afcd8d4"}
Jul 30 02:59:19.839598 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:19.839577    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4fcfd02b-kx2ff" event={"ID":"a58ae441-7361-4999-a8dc-ee5f8b2b9d57","Type":"ContainerStarted","Data":"8533b065a1989a26ecf2cf881e8032bb1a045e4000b74396336045d7aba086aa"}
Jul 30 02:59:19.859723 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:19.859671    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4fcfd02b-kx2ff" podStartSLOduration=1.859654698 podStartE2EDuration="1.859654698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:59:19.857233004 +0000 UTC m=+1995.504016406" watchObservedRunningTime="2026-07-30 02:59:19.859654698 +0000 UTC m=+1995.506438102"
Jul 30 02:59:20.846679 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:20.846647    2565 generic.go:356] "Generic (PLEG): container finished" podID="a58ae441-7361-4999-a8dc-ee5f8b2b9d57" containerID="4d8443a1a18ffe3142a4180221bef139e3d21d675a773df7971e0b474afcd8d4" exitCode=0
Jul 30 02:59:20.847068 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:20.846711    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4fcfd02b-kx2ff" event={"ID":"a58ae441-7361-4999-a8dc-ee5f8b2b9d57","Type":"ContainerDied","Data":"4d8443a1a18ffe3142a4180221bef139e3d21d675a773df7971e0b474afcd8d4"}
Jul 30 02:59:22.003057 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:22.003026    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4fcfd02b-kx2ff"
Jul 30 02:59:22.051291 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:22.051258    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58ae441-7361-4999-a8dc-ee5f8b2b9d57-maas-api-service-ca\") pod \"a58ae441-7361-4999-a8dc-ee5f8b2b9d57\" (UID: \"a58ae441-7361-4999-a8dc-ee5f8b2b9d57\") "
Jul 30 02:59:22.051671 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:22.051645    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a58ae441-7361-4999-a8dc-ee5f8b2b9d57-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a58ae441-7361-4999-a8dc-ee5f8b2b9d57" (UID: "a58ae441-7361-4999-a8dc-ee5f8b2b9d57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 02:59:22.051751 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:22.051716    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a58ae441-7361-4999-a8dc-ee5f8b2b9d57-maas-api-service-ca\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:22.859837 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:22.859804    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4fcfd02b-kx2ff" event={"ID":"a58ae441-7361-4999-a8dc-ee5f8b2b9d57","Type":"ContainerDied","Data":"8533b065a1989a26ecf2cf881e8032bb1a045e4000b74396336045d7aba086aa"}
Jul 30 02:59:22.859993 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:22.859837    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4fcfd02b-kx2ff"
Jul 30 02:59:22.859993 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:22.859855    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8533b065a1989a26ecf2cf881e8032bb1a045e4000b74396336045d7aba086aa"
Jul 30 02:59:23.588813 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:23.588784    2565 patch_prober.go:28] interesting pod/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": read tcp 10.134.0.2:36616->10.134.0.63:15021: read: connection reset by peer" start-of-body=
Jul 30 02:59:23.589198 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:23.588849    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9" podUID="e866047e-a275-4432-9476-19f88302a32c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": read tcp 10.134.0.2:36616->10.134.0.63:15021: read: connection reset by peer"
Jul 30 02:59:23.867237 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:23.867139    2565 generic.go:356] "Generic (PLEG): container finished" podID="e866047e-a275-4432-9476-19f88302a32c" containerID="fd70eae7f0bc518cf9e61e7feee7191078161b9a4209a5674e9453fe63dc6ccf" exitCode=0
Jul 30 02:59:23.867237 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:23.867205    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9" event={"ID":"e866047e-a275-4432-9476-19f88302a32c","Type":"ContainerDied","Data":"fd70eae7f0bc518cf9e61e7feee7191078161b9a4209a5674e9453fe63dc6ccf"}
Jul 30 02:59:23.867569 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:23.867555    2565 scope.go:117] "RemoveContainer" containerID="fd70eae7f0bc518cf9e61e7feee7191078161b9a4209a5674e9453fe63dc6ccf"
Jul 30 02:59:23.868733 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:23.868679    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:59:23.868818 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:23.868771    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:59:23.868889 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:23.868831    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:59:24.876291 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:24.876255    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9" event={"ID":"e866047e-a275-4432-9476-19f88302a32c","Type":"ContainerStarted","Data":"c924769b1095832ba07c8acba464aa7c3ade110baf8ebbacbb800bffac96e77b"}
Jul 30 02:59:25.201250 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:25.201133    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:25.201513 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:25.201450    2565 patch_prober.go:28] interesting pod/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Jul 30 02:59:25.201601 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:25.201529    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9" podUID="e866047e-a275-4432-9476-19f88302a32c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Jul 30 02:59:26.201408 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:26.201364    2565 patch_prober.go:28] interesting pod/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Jul 30 02:59:26.201918 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:26.201433    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9" podUID="e866047e-a275-4432-9476-19f88302a32c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Jul 30 02:59:27.201347 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:27.201312    2565 patch_prober.go:28] interesting pod/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Jul 30 02:59:27.201551 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:27.201372    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9" podUID="e866047e-a275-4432-9476-19f88302a32c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Jul 30 02:59:28.201521 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:28.201486    2565 patch_prober.go:28] interesting pod/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Jul 30 02:59:28.201706 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:28.201544    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9" podUID="e866047e-a275-4432-9476-19f88302a32c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Jul 30 02:59:29.204779 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.204745    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:29.205294 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.205275    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:29.206043 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.206013    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:29.282541 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.282503    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"]
Jul 30 02:59:29.282832 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.282799    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm" podUID="e7150223-1c12-4051-925e-533b69a63e1a" containerName="maas-api" containerID="cri-o://2c3bbb10fc113978206f8edb9d0587d126f98410d460d99ead89c3a81a265b5b" gracePeriod=30
Jul 30 02:59:29.547523 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.547497    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"
Jul 30 02:59:29.625606 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.625511    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7150223-1c12-4051-925e-533b69a63e1a-openshift-service-ca\") pod \"e7150223-1c12-4051-925e-533b69a63e1a\" (UID: \"e7150223-1c12-4051-925e-533b69a63e1a\") "
Jul 30 02:59:29.625796 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.625639    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djhqz\" (UniqueName: \"kubernetes.io/projected/e7150223-1c12-4051-925e-533b69a63e1a-kube-api-access-djhqz\") pod \"e7150223-1c12-4051-925e-533b69a63e1a\" (UID: \"e7150223-1c12-4051-925e-533b69a63e1a\") "
Jul 30 02:59:29.625796 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.625734    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7150223-1c12-4051-925e-533b69a63e1a-maas-api-tls\") pod \"e7150223-1c12-4051-925e-533b69a63e1a\" (UID: \"e7150223-1c12-4051-925e-533b69a63e1a\") "
Jul 30 02:59:29.625935 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.625883    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7150223-1c12-4051-925e-533b69a63e1a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e7150223-1c12-4051-925e-533b69a63e1a" (UID: "e7150223-1c12-4051-925e-533b69a63e1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 02:59:29.626428 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.626400    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7150223-1c12-4051-925e-533b69a63e1a-openshift-service-ca\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:29.628329 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.628302    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7150223-1c12-4051-925e-533b69a63e1a-kube-api-access-djhqz" (OuterVolumeSpecName: "kube-api-access-djhqz") pod "e7150223-1c12-4051-925e-533b69a63e1a" (UID: "e7150223-1c12-4051-925e-533b69a63e1a"). InnerVolumeSpecName "kube-api-access-djhqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:59:29.628771 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.628737    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7150223-1c12-4051-925e-533b69a63e1a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e7150223-1c12-4051-925e-533b69a63e1a" (UID: "e7150223-1c12-4051-925e-533b69a63e1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 02:59:29.727790 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.727706    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djhqz\" (UniqueName: \"kubernetes.io/projected/e7150223-1c12-4051-925e-533b69a63e1a-kube-api-access-djhqz\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:29.727790 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.727746    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7150223-1c12-4051-925e-533b69a63e1a-maas-api-tls\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:29.914421 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.914383    2565 generic.go:356] "Generic (PLEG): container finished" podID="e7150223-1c12-4051-925e-533b69a63e1a" containerID="2c3bbb10fc113978206f8edb9d0587d126f98410d460d99ead89c3a81a265b5b" exitCode=0
Jul 30 02:59:29.914587 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.914433    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"
Jul 30 02:59:29.914587 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.914475    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm" event={"ID":"e7150223-1c12-4051-925e-533b69a63e1a","Type":"ContainerDied","Data":"2c3bbb10fc113978206f8edb9d0587d126f98410d460d99ead89c3a81a265b5b"}
Jul 30 02:59:29.914587 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.914537    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm" event={"ID":"e7150223-1c12-4051-925e-533b69a63e1a","Type":"ContainerDied","Data":"450997f88e3c5af89852c9fa11b1206000e3444f65285948aa2576cf71ec9df0"}
Jul 30 02:59:29.914587 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.914574    2565 scope.go:117] "RemoveContainer" containerID="2c3bbb10fc113978206f8edb9d0587d126f98410d460d99ead89c3a81a265b5b"
Jul 30 02:59:29.924518 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.924499    2565 scope.go:117] "RemoveContainer" containerID="2c3bbb10fc113978206f8edb9d0587d126f98410d460d99ead89c3a81a265b5b"
Jul 30 02:59:29.924795 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:59:29.924775    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c3bbb10fc113978206f8edb9d0587d126f98410d460d99ead89c3a81a265b5b\": container with ID starting with 2c3bbb10fc113978206f8edb9d0587d126f98410d460d99ead89c3a81a265b5b not found: ID does not exist" containerID="2c3bbb10fc113978206f8edb9d0587d126f98410d460d99ead89c3a81a265b5b"
Jul 30 02:59:29.924844 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.924804    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c3bbb10fc113978206f8edb9d0587d126f98410d460d99ead89c3a81a265b5b"} err="failed to get container status \"2c3bbb10fc113978206f8edb9d0587d126f98410d460d99ead89c3a81a265b5b\": rpc error: code = NotFound desc = could not find container \"2c3bbb10fc113978206f8edb9d0587d126f98410d460d99ead89c3a81a265b5b\": container with ID starting with 2c3bbb10fc113978206f8edb9d0587d126f98410d460d99ead89c3a81a265b5b not found: ID does not exist"
Jul 30 02:59:29.936332 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.936301    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"]
Jul 30 02:59:29.942688 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:29.942669    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4fcfd02b-7557dcc6c-9rxjm"]
Jul 30 02:59:30.802056 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:30.802022    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7150223-1c12-4051-925e-533b69a63e1a" path="/var/lib/kubelet/pods/e7150223-1c12-4051-925e-533b69a63e1a/volumes"
Jul 30 02:59:33.567479 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:33.567444    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4fcfd02b-kx2ff"]
Jul 30 02:59:33.569697 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:33.569672    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4fcfd02b-kx2ff"]
Jul 30 02:59:34.718841 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:34.718807    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"]
Jul 30 02:59:34.719254 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:34.719084    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9" podUID="e866047e-a275-4432-9476-19f88302a32c" containerName="istio-proxy" containerID="cri-o://c924769b1095832ba07c8acba464aa7c3ade110baf8ebbacbb800bffac96e77b" gracePeriod=30
Jul 30 02:59:34.801626 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:34.801591    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a58ae441-7361-4999-a8dc-ee5f8b2b9d57" path="/var/lib/kubelet/pods/a58ae441-7361-4999-a8dc-ee5f8b2b9d57/volumes"
Jul 30 02:59:35.415235 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.415203    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"]
Jul 30 02:59:35.415656 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.415638    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7150223-1c12-4051-925e-533b69a63e1a" containerName="maas-api"
Jul 30 02:59:35.415818 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.415658    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7150223-1c12-4051-925e-533b69a63e1a" containerName="maas-api"
Jul 30 02:59:35.415818 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.415685    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58ae441-7361-4999-a8dc-ee5f8b2b9d57" containerName="revoke-keys"
Jul 30 02:59:35.415818 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.415694    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58ae441-7361-4999-a8dc-ee5f8b2b9d57" containerName="revoke-keys"
Jul 30 02:59:35.415818 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.415817    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7150223-1c12-4051-925e-533b69a63e1a" containerName="maas-api"
Jul 30 02:59:35.416023 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.415832    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a58ae441-7361-4999-a8dc-ee5f8b2b9d57" containerName="revoke-keys"
Jul 30 02:59:35.420306 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.420287    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.423477 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.423455    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-1ca4664c-openshift-default-dockercfg-2wfh6\""
Jul 30 02:59:35.432768 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.432741    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"]
Jul 30 02:59:35.584678 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.584643    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-workload-certs\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.584678 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.584678    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-data\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.584943 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.584699    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npt4w\" (UniqueName: \"kubernetes.io/projected/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-kube-api-access-npt4w\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.584943 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.584722    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-workload-socket\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.584943 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.584747    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-envoy\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.584943 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.584774    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-podinfo\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.584943 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.584816    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-token\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.584943 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.584833    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-credential-socket\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.584943 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.584866    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istiod-ca-cert\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.686407 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.686317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istiod-ca-cert\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.686407 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.686385    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-workload-certs\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.686407 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.686403    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-data\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.686713 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.686420    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npt4w\" (UniqueName: \"kubernetes.io/projected/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-kube-api-access-npt4w\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.686713 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.686440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-workload-socket\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.686713 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.686457    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-envoy\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.686713 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.686488    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-podinfo\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.686713 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.686540    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-token\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.686713 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.686568    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-credential-socket\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.687034 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.686887    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-workload-certs\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.687034 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.686933    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-credential-socket\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.687248 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.687220    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istiod-ca-cert\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.687248 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.687233    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-data\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.687411 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.687292    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-workload-socket\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.689157 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.689138    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-envoy\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.689578 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.689549    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-podinfo\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.695334 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.695312    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npt4w\" (UniqueName: \"kubernetes.io/projected/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-kube-api-access-npt4w\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.695528 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.695509    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-token\") pod \"e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") " pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.734352 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.734325    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:35.879772 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.879748    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"]
Jul 30 02:59:35.882413 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:59:35.882385    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd3f4fa_3ae5_4991_9523_950cb412bfc9.slice/crio-6cdb09cf3e0c3efc049b5c9805aeced0bbd659b1f5a6ddb6fb5f65f86a197dbf WatchSource:0}: Error finding container 6cdb09cf3e0c3efc049b5c9805aeced0bbd659b1f5a6ddb6fb5f65f86a197dbf: Status 404 returned error can't find the container with id 6cdb09cf3e0c3efc049b5c9805aeced0bbd659b1f5a6ddb6fb5f65f86a197dbf
Jul 30 02:59:35.884838 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.884807    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:59:35.884931 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.884868    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:59:35.884931 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.884897    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:59:35.949943 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:35.949918    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx" event={"ID":"ccd3f4fa-3ae5-4991-9523-950cb412bfc9","Type":"ContainerStarted","Data":"6cdb09cf3e0c3efc049b5c9805aeced0bbd659b1f5a6ddb6fb5f65f86a197dbf"}
Jul 30 02:59:36.957585 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:36.957549    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx" event={"ID":"ccd3f4fa-3ae5-4991-9523-950cb412bfc9","Type":"ContainerStarted","Data":"cad5263dd86af6480d422245c6cc08a57d0f4261935297867aae4ec2ea4d7a01"}
Jul 30 02:59:36.979378 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:36.979335    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx" podStartSLOduration=1.9793206840000002 podStartE2EDuration="1.979320684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:59:36.976941793 +0000 UTC m=+2012.623725200" watchObservedRunningTime="2026-07-30 02:59:36.979320684 +0000 UTC m=+2012.626104089"
Jul 30 02:59:37.735484 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:37.735449    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:37.740534 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:37.740513    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:37.963840 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:37.963813    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:37.964842 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:37.964822    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:38.553579 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:38.553543    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"]
Jul 30 02:59:38.557117 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:38.557096    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"
Jul 30 02:59:38.559825 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:38.559804    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-1ca4664c-serving-cert\""
Jul 30 02:59:38.570641 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:38.570603    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"]
Jul 30 02:59:38.715891 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:38.715855    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52713660-da7d-408b-832e-d33f9e298e10-openshift-service-ca\") pod \"maas-api-e2e-derive-1ca4664c-79df65d49-nblvm\" (UID: \"52713660-da7d-408b-832e-d33f9e298e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"
Jul 30 02:59:38.716077 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:38.715911    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52713660-da7d-408b-832e-d33f9e298e10-maas-api-tls\") pod \"maas-api-e2e-derive-1ca4664c-79df65d49-nblvm\" (UID: \"52713660-da7d-408b-832e-d33f9e298e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"
Jul 30 02:59:38.716077 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:38.715938    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzldk\" (UniqueName: \"kubernetes.io/projected/52713660-da7d-408b-832e-d33f9e298e10-kube-api-access-gzldk\") pod \"maas-api-e2e-derive-1ca4664c-79df65d49-nblvm\" (UID: \"52713660-da7d-408b-832e-d33f9e298e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"
Jul 30 02:59:38.817130 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:38.817046    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52713660-da7d-408b-832e-d33f9e298e10-openshift-service-ca\") pod \"maas-api-e2e-derive-1ca4664c-79df65d49-nblvm\" (UID: \"52713660-da7d-408b-832e-d33f9e298e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"
Jul 30 02:59:38.817130 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:38.817102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52713660-da7d-408b-832e-d33f9e298e10-maas-api-tls\") pod \"maas-api-e2e-derive-1ca4664c-79df65d49-nblvm\" (UID: \"52713660-da7d-408b-832e-d33f9e298e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"
Jul 30 02:59:38.817130 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:38.817120    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzldk\" (UniqueName: \"kubernetes.io/projected/52713660-da7d-408b-832e-d33f9e298e10-kube-api-access-gzldk\") pod \"maas-api-e2e-derive-1ca4664c-79df65d49-nblvm\" (UID: \"52713660-da7d-408b-832e-d33f9e298e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"
Jul 30 02:59:38.817689 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:38.817657    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52713660-da7d-408b-832e-d33f9e298e10-openshift-service-ca\") pod \"maas-api-e2e-derive-1ca4664c-79df65d49-nblvm\" (UID: \"52713660-da7d-408b-832e-d33f9e298e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"
Jul 30 02:59:38.819656 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:38.819633    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52713660-da7d-408b-832e-d33f9e298e10-maas-api-tls\") pod \"maas-api-e2e-derive-1ca4664c-79df65d49-nblvm\" (UID: \"52713660-da7d-408b-832e-d33f9e298e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"
Jul 30 02:59:38.825739 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:38.825719    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzldk\" (UniqueName: \"kubernetes.io/projected/52713660-da7d-408b-832e-d33f9e298e10-kube-api-access-gzldk\") pod \"maas-api-e2e-derive-1ca4664c-79df65d49-nblvm\" (UID: \"52713660-da7d-408b-832e-d33f9e298e10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"
Jul 30 02:59:38.868952 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:38.868923    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"
Jul 30 02:59:39.014711 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.014678    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"]
Jul 30 02:59:39.015678 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:59:39.015651    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52713660_da7d_408b_832e_d33f9e298e10.slice/crio-0e1fdc5ae0bd7c813743d8ac52fceef72286bcd60e6ac70e028d1ebf9448be33 WatchSource:0}: Error finding container 0e1fdc5ae0bd7c813743d8ac52fceef72286bcd60e6ac70e028d1ebf9448be33: Status 404 returned error can't find the container with id 0e1fdc5ae0bd7c813743d8ac52fceef72286bcd60e6ac70e028d1ebf9448be33
Jul 30 02:59:39.219423 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.219327    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd"]
Jul 30 02:59:39.223985 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.223964    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd"
Jul 30 02:59:39.226662 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.226642    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-1ca4664c-dockercfg-jhdmv\""
Jul 30 02:59:39.226779 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.226685    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-1ca4664c\""
Jul 30 02:59:39.232496 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.232234    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd"]
Jul 30 02:59:39.321998 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.321965    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ea31c03-28fa-43e9-af77-5539bef9e23c-plugins-config-volume\") pod \"payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd\" (UID: \"6ea31c03-28fa-43e9-af77-5539bef9e23c\") " pod="openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd"
Jul 30 02:59:39.322188 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.322100    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq5fj\" (UniqueName: \"kubernetes.io/projected/6ea31c03-28fa-43e9-af77-5539bef9e23c-kube-api-access-tq5fj\") pod \"payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd\" (UID: \"6ea31c03-28fa-43e9-af77-5539bef9e23c\") " pod="openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd"
Jul 30 02:59:39.423353 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.423329    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq5fj\" (UniqueName: \"kubernetes.io/projected/6ea31c03-28fa-43e9-af77-5539bef9e23c-kube-api-access-tq5fj\") pod \"payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd\" (UID: \"6ea31c03-28fa-43e9-af77-5539bef9e23c\") " pod="openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd"
Jul 30 02:59:39.423472 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.423380    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ea31c03-28fa-43e9-af77-5539bef9e23c-plugins-config-volume\") pod \"payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd\" (UID: \"6ea31c03-28fa-43e9-af77-5539bef9e23c\") " pod="openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd"
Jul 30 02:59:39.424258 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.424235    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ea31c03-28fa-43e9-af77-5539bef9e23c-plugins-config-volume\") pod \"payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd\" (UID: \"6ea31c03-28fa-43e9-af77-5539bef9e23c\") " pod="openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd"
Jul 30 02:59:39.431608 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.431582    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq5fj\" (UniqueName: \"kubernetes.io/projected/6ea31c03-28fa-43e9-af77-5539bef9e23c-kube-api-access-tq5fj\") pod \"payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd\" (UID: \"6ea31c03-28fa-43e9-af77-5539bef9e23c\") " pod="openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd"
Jul 30 02:59:39.536252 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.536148    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd"
Jul 30 02:59:39.680087 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.680039    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd"]
Jul 30 02:59:39.683090 ip-10-0-132-89 kubenswrapper[2565]: W0730 02:59:39.683057    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea31c03_28fa_43e9_af77_5539bef9e23c.slice/crio-995b795ef0c706be9c963e03dbc608649fe3ed80591de5a674192eb09c3e45ac WatchSource:0}: Error finding container 995b795ef0c706be9c963e03dbc608649fe3ed80591de5a674192eb09c3e45ac: Status 404 returned error can't find the container with id 995b795ef0c706be9c963e03dbc608649fe3ed80591de5a674192eb09c3e45ac
Jul 30 02:59:39.982034 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.982003    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm" event={"ID":"52713660-da7d-408b-832e-d33f9e298e10","Type":"ContainerStarted","Data":"f79754e2bc3bc8ad8d947a8c7cefcf97bdc3802d2871fa622a453c386e1c188e"}
Jul 30 02:59:39.982215 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.982066    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm" event={"ID":"52713660-da7d-408b-832e-d33f9e298e10","Type":"ContainerStarted","Data":"0e1fdc5ae0bd7c813743d8ac52fceef72286bcd60e6ac70e028d1ebf9448be33"}
Jul 30 02:59:39.982215 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.982117    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"
Jul 30 02:59:39.983155 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.983136    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:39.983840 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.983818    2565 generic.go:356] "Generic (PLEG): container finished" podID="e866047e-a275-4432-9476-19f88302a32c" containerID="c924769b1095832ba07c8acba464aa7c3ade110baf8ebbacbb800bffac96e77b" exitCode=0
Jul 30 02:59:39.983945 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.983889    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9" event={"ID":"e866047e-a275-4432-9476-19f88302a32c","Type":"ContainerDied","Data":"c924769b1095832ba07c8acba464aa7c3ade110baf8ebbacbb800bffac96e77b"}
Jul 30 02:59:39.983945 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.983935    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9" event={"ID":"e866047e-a275-4432-9476-19f88302a32c","Type":"ContainerDied","Data":"74bc4fb7817b7055ccd7674442b659f4b863a26bca216c1a65dbcdd0b0db9acf"}
Jul 30 02:59:39.984051 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.983970    2565 scope.go:117] "RemoveContainer" containerID="c924769b1095832ba07c8acba464aa7c3ade110baf8ebbacbb800bffac96e77b"
Jul 30 02:59:39.985956 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.985936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd" event={"ID":"6ea31c03-28fa-43e9-af77-5539bef9e23c","Type":"ContainerStarted","Data":"d74332cab9360d23915039db5a64f0f28d6121ed66cc86e9ed26a7f7bb10de37"}
Jul 30 02:59:39.986045 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.985980    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd" event={"ID":"6ea31c03-28fa-43e9-af77-5539bef9e23c","Type":"ContainerStarted","Data":"995b795ef0c706be9c963e03dbc608649fe3ed80591de5a674192eb09c3e45ac"}
Jul 30 02:59:39.986045 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.986037    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd"
Jul 30 02:59:39.998938 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.998914    2565 scope.go:117] "RemoveContainer" containerID="fd70eae7f0bc518cf9e61e7feee7191078161b9a4209a5674e9453fe63dc6ccf"
Jul 30 02:59:39.999066 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:39.998950    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm" podStartSLOduration=1.649740872 podStartE2EDuration="1.998931815s" totalImagesPullingTime="349.190943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:59:39.998041682 +0000 UTC m=+2015.644825089" watchObservedRunningTime="2026-07-30 02:59:39.998931815 +0000 UTC m=+2015.645715222"
Jul 30 02:59:40.008269 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.008250    2565 scope.go:117] "RemoveContainer" containerID="c924769b1095832ba07c8acba464aa7c3ade110baf8ebbacbb800bffac96e77b"
Jul 30 02:59:40.008508 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:59:40.008490    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c924769b1095832ba07c8acba464aa7c3ade110baf8ebbacbb800bffac96e77b\": container with ID starting with c924769b1095832ba07c8acba464aa7c3ade110baf8ebbacbb800bffac96e77b not found: ID does not exist" containerID="c924769b1095832ba07c8acba464aa7c3ade110baf8ebbacbb800bffac96e77b"
Jul 30 02:59:40.008573 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.008515    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c924769b1095832ba07c8acba464aa7c3ade110baf8ebbacbb800bffac96e77b"} err="failed to get container status \"c924769b1095832ba07c8acba464aa7c3ade110baf8ebbacbb800bffac96e77b\": rpc error: code = NotFound desc = could not find container \"c924769b1095832ba07c8acba464aa7c3ade110baf8ebbacbb800bffac96e77b\": container with ID starting with c924769b1095832ba07c8acba464aa7c3ade110baf8ebbacbb800bffac96e77b not found: ID does not exist"
Jul 30 02:59:40.008573 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.008530    2565 scope.go:117] "RemoveContainer" containerID="fd70eae7f0bc518cf9e61e7feee7191078161b9a4209a5674e9453fe63dc6ccf"
Jul 30 02:59:40.008773 ip-10-0-132-89 kubenswrapper[2565]: E0730 02:59:40.008759    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd70eae7f0bc518cf9e61e7feee7191078161b9a4209a5674e9453fe63dc6ccf\": container with ID starting with fd70eae7f0bc518cf9e61e7feee7191078161b9a4209a5674e9453fe63dc6ccf not found: ID does not exist" containerID="fd70eae7f0bc518cf9e61e7feee7191078161b9a4209a5674e9453fe63dc6ccf"
Jul 30 02:59:40.008856 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.008776    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd70eae7f0bc518cf9e61e7feee7191078161b9a4209a5674e9453fe63dc6ccf"} err="failed to get container status \"fd70eae7f0bc518cf9e61e7feee7191078161b9a4209a5674e9453fe63dc6ccf\": rpc error: code = NotFound desc = could not find container \"fd70eae7f0bc518cf9e61e7feee7191078161b9a4209a5674e9453fe63dc6ccf\": container with ID starting with fd70eae7f0bc518cf9e61e7feee7191078161b9a4209a5674e9453fe63dc6ccf not found: ID does not exist"
Jul 30 02:59:40.016825 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.016777    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd" podStartSLOduration=1.016758891 podStartE2EDuration="1.016758891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 02:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 02:59:40.013100118 +0000 UTC m=+2015.659883523" watchObservedRunningTime="2026-07-30 02:59:40.016758891 +0000 UTC m=+2015.663542297"
Jul 30 02:59:40.131480 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.131401    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-istio-envoy\") pod \"e866047e-a275-4432-9476-19f88302a32c\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") "
Jul 30 02:59:40.131630 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.131500    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-workload-socket\") pod \"e866047e-a275-4432-9476-19f88302a32c\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") "
Jul 30 02:59:40.131630 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.131573    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e866047e-a275-4432-9476-19f88302a32c-istiod-ca-cert\") pod \"e866047e-a275-4432-9476-19f88302a32c\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") "
Jul 30 02:59:40.131630 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.131597    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-workload-certs\") pod \"e866047e-a275-4432-9476-19f88302a32c\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") "
Jul 30 02:59:40.131792 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.131627    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pf47\" (UniqueName: \"kubernetes.io/projected/e866047e-a275-4432-9476-19f88302a32c-kube-api-access-2pf47\") pod \"e866047e-a275-4432-9476-19f88302a32c\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") "
Jul 30 02:59:40.131792 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.131745    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e866047e-a275-4432-9476-19f88302a32c" (UID: "e866047e-a275-4432-9476-19f88302a32c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:59:40.131792 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.131754    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e866047e-a275-4432-9476-19f88302a32c-istio-podinfo\") pod \"e866047e-a275-4432-9476-19f88302a32c\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") "
Jul 30 02:59:40.131959 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.131798    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e866047e-a275-4432-9476-19f88302a32c-istio-token\") pod \"e866047e-a275-4432-9476-19f88302a32c\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") "
Jul 30 02:59:40.131959 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.131841    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e866047e-a275-4432-9476-19f88302a32c" (UID: "e866047e-a275-4432-9476-19f88302a32c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:59:40.131959 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.131848    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-istio-data\") pod \"e866047e-a275-4432-9476-19f88302a32c\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") "
Jul 30 02:59:40.131959 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.131897    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-credential-socket\") pod \"e866047e-a275-4432-9476-19f88302a32c\" (UID: \"e866047e-a275-4432-9476-19f88302a32c\") "
Jul 30 02:59:40.132195 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.132077    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e866047e-a275-4432-9476-19f88302a32c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e866047e-a275-4432-9476-19f88302a32c" (UID: "e866047e-a275-4432-9476-19f88302a32c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 02:59:40.132319 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.132289    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e866047e-a275-4432-9476-19f88302a32c" (UID: "e866047e-a275-4432-9476-19f88302a32c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:59:40.132482 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.132456    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-credential-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:40.132482 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.132478    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-workload-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:40.132673 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.132492    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e866047e-a275-4432-9476-19f88302a32c-istiod-ca-cert\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:40.132673 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.132505    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-workload-certs\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:40.132849 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.132820    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-istio-data" (OuterVolumeSpecName: "istio-data") pod "e866047e-a275-4432-9476-19f88302a32c" (UID: "e866047e-a275-4432-9476-19f88302a32c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:59:40.133998 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.133971    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e866047e-a275-4432-9476-19f88302a32c" (UID: "e866047e-a275-4432-9476-19f88302a32c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 02:59:40.134751 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.134728    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e866047e-a275-4432-9476-19f88302a32c-kube-api-access-2pf47" (OuterVolumeSpecName: "kube-api-access-2pf47") pod "e866047e-a275-4432-9476-19f88302a32c" (UID: "e866047e-a275-4432-9476-19f88302a32c"). InnerVolumeSpecName "kube-api-access-2pf47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:59:40.134926 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.134904    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e866047e-a275-4432-9476-19f88302a32c-istio-token" (OuterVolumeSpecName: "istio-token") pod "e866047e-a275-4432-9476-19f88302a32c" (UID: "e866047e-a275-4432-9476-19f88302a32c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 02:59:40.135379 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.135362    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e866047e-a275-4432-9476-19f88302a32c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e866047e-a275-4432-9476-19f88302a32c" (UID: "e866047e-a275-4432-9476-19f88302a32c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 30 02:59:40.233908 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.233877    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-istio-envoy\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:40.233908 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.233904    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pf47\" (UniqueName: \"kubernetes.io/projected/e866047e-a275-4432-9476-19f88302a32c-kube-api-access-2pf47\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:40.234099 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.233916    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e866047e-a275-4432-9476-19f88302a32c-istio-podinfo\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:40.234099 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.233929    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e866047e-a275-4432-9476-19f88302a32c-istio-token\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:40.234099 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.233940    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e866047e-a275-4432-9476-19f88302a32c-istio-data\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 02:59:40.992823 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:40.992792    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"
Jul 30 02:59:41.015672 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:41.015630    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"]
Jul 30 02:59:41.020753 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:41.020731    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4fcfd02b-openshift-default-848854b774-7khx9"]
Jul 30 02:59:42.801101 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:42.801057    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e866047e-a275-4432-9476-19f88302a32c" path="/var/lib/kubelet/pods/e866047e-a275-4432-9476-19f88302a32c/volumes"
Jul 30 02:59:45.998647 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:45.998622    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"
Jul 30 02:59:50.995638 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:50.995609    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd"
Jul 30 02:59:53.735495 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:53.735458    2565 patch_prober.go:28] interesting pod/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": read tcp 10.134.0.2:54236->10.134.0.66:15021: read: connection reset by peer" start-of-body=
Jul 30 02:59:53.735897 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:53.735538    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx" podUID="ccd3f4fa-3ae5-4991-9523-950cb412bfc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": read tcp 10.134.0.2:54236->10.134.0.66:15021: read: connection reset by peer"
Jul 30 02:59:54.074596 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:54.074552    2565 generic.go:356] "Generic (PLEG): container finished" podID="ccd3f4fa-3ae5-4991-9523-950cb412bfc9" containerID="cad5263dd86af6480d422245c6cc08a57d0f4261935297867aae4ec2ea4d7a01" exitCode=0
Jul 30 02:59:54.074779 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:54.074621    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx" event={"ID":"ccd3f4fa-3ae5-4991-9523-950cb412bfc9","Type":"ContainerDied","Data":"cad5263dd86af6480d422245c6cc08a57d0f4261935297867aae4ec2ea4d7a01"}
Jul 30 02:59:54.075064 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:54.075048    2565 scope.go:117] "RemoveContainer" containerID="cad5263dd86af6480d422245c6cc08a57d0f4261935297867aae4ec2ea4d7a01"
Jul 30 02:59:54.076115 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:54.076086    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:59:54.076244 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:54.076187    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:59:54.076324 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:54.076243    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 02:59:55.087198 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:55.086763    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx" event={"ID":"ccd3f4fa-3ae5-4991-9523-950cb412bfc9","Type":"ContainerStarted","Data":"52b9f176cd83345c955740113bffffbb41eb3c48aadd52be518bc3d01f782ee1"}
Jul 30 02:59:55.735085 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:55.735046    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:55.735343 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:55.735282    2565 patch_prober.go:28] interesting pod/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 30 02:59:55.735429 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:55.735340    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx" podUID="ccd3f4fa-3ae5-4991-9523-950cb412bfc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 30 02:59:56.735656 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:56.735619    2565 patch_prober.go:28] interesting pod/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 30 02:59:56.736135 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:56.735680    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx" podUID="ccd3f4fa-3ae5-4991-9523-950cb412bfc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 30 02:59:57.734982 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:57.734947    2565 patch_prober.go:28] interesting pod/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 30 02:59:57.735201 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:57.735005    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx" podUID="ccd3f4fa-3ae5-4991-9523-950cb412bfc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 30 02:59:58.735144 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:58.735107    2565 patch_prober.go:28] interesting pod/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 30 02:59:58.735557 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:58.735206    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx" podUID="ccd3f4fa-3ae5-4991-9523-950cb412bfc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 30 02:59:59.738058 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:59.738028    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:59.738510 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:59.738373    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 02:59:59.739118 ip-10-0-132-89 kubenswrapper[2565]: I0730 02:59:59.739098    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 03:00:00.130247 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:00.130219    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756340-42grx"]
Jul 30 03:00:00.130798 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:00.130772    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e866047e-a275-4432-9476-19f88302a32c" containerName="istio-proxy"
Jul 30 03:00:00.130798 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:00.130799    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e866047e-a275-4432-9476-19f88302a32c" containerName="istio-proxy"
Jul 30 03:00:00.130950 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:00.130826    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e866047e-a275-4432-9476-19f88302a32c" containerName="istio-proxy"
Jul 30 03:00:00.130950 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:00.130835    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e866047e-a275-4432-9476-19f88302a32c" containerName="istio-proxy"
Jul 30 03:00:00.130950 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:00.130928    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e866047e-a275-4432-9476-19f88302a32c" containerName="istio-proxy"
Jul 30 03:00:00.133121 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:00.133094    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1ca4664c-29756340-sfdg8"]
Jul 30 03:00:00.133331 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:00.133109    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756340-42grx"
Jul 30 03:00:00.133592 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:00.133577    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e866047e-a275-4432-9476-19f88302a32c" containerName="istio-proxy"
Jul 30 03:00:00.135342 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:00.135327    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1ca4664c-29756340-sfdg8"
Jul 30 03:00:00.135810 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:00.135796    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2npk7\""
Jul 30 03:00:00.142288 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:00.142264    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756340-42grx"]
Jul 30 03:00:00.145726 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:00.145705    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1ca4664c-29756340-sfdg8"]
Jul 30 03:00:00.149969 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:00.149950    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756340-42grx"
Jul 30 03:00:00.157038 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:00.157021    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1ca4664c-29756340-sfdg8"
Jul 30 03:00:00.314119 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:00.314041    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756340-42grx"]
Jul 30 03:00:00.314369 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:00:00.314341    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a8e67e6_6266_4420_82b7_4f70e0678a9f.slice/crio-f89294ef4f1b091c5920d119f287a960f1a4d6a7d8c050c3eff5dff4874985e9 WatchSource:0}: Error finding container f89294ef4f1b091c5920d119f287a960f1a4d6a7d8c050c3eff5dff4874985e9: Status 404 returned error can't find the container with id f89294ef4f1b091c5920d119f287a960f1a4d6a7d8c050c3eff5dff4874985e9
Jul 30 03:00:00.349849 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:00.349826    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1ca4664c-29756340-sfdg8"]
Jul 30 03:00:00.355292 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:00:00.355263    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24701fca_d6a5_4c4e_8126_cd0c773b7aef.slice/crio-2c57db896d574bbffec7e96ee6ed3508bd49f01c89bc2604cbe7229b14f4ae1f WatchSource:0}: Error finding container 2c57db896d574bbffec7e96ee6ed3508bd49f01c89bc2604cbe7229b14f4ae1f: Status 404 returned error can't find the container with id 2c57db896d574bbffec7e96ee6ed3508bd49f01c89bc2604cbe7229b14f4ae1f
Jul 30 03:00:01.129179 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:01.129119    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756340-42grx" event={"ID":"6a8e67e6-6266-4420-82b7-4f70e0678a9f","Type":"ContainerStarted","Data":"fa32286423813dc4fe99dfcc5906e3115a255155dc6bcaca9d97787f618d1f5b"}
Jul 30 03:00:01.129605 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:01.129215    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756340-42grx" event={"ID":"6a8e67e6-6266-4420-82b7-4f70e0678a9f","Type":"ContainerStarted","Data":"f89294ef4f1b091c5920d119f287a960f1a4d6a7d8c050c3eff5dff4874985e9"}
Jul 30 03:00:01.130930 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:01.130897    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1ca4664c-29756340-sfdg8" event={"ID":"24701fca-d6a5-4c4e-8126-cd0c773b7aef","Type":"ContainerStarted","Data":"232c140f8ffc80e492a6eb7106e758859e4c93718b5a812451a0a616b2a8188f"}
Jul 30 03:00:01.131055 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:01.130946    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1ca4664c-29756340-sfdg8" event={"ID":"24701fca-d6a5-4c4e-8126-cd0c773b7aef","Type":"ContainerStarted","Data":"2c57db896d574bbffec7e96ee6ed3508bd49f01c89bc2604cbe7229b14f4ae1f"}
Jul 30 03:00:01.146101 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:01.146054    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756340-42grx" podStartSLOduration=1.1460406380000001 podStartE2EDuration="1.146040638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:00:01.144183149 +0000 UTC m=+2036.790966544" watchObservedRunningTime="2026-07-30 03:00:01.146040638 +0000 UTC m=+2036.792824080"
Jul 30 03:00:01.159077 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:01.159032    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1ca4664c-29756340-sfdg8" podStartSLOduration=1.159018706 podStartE2EDuration="1.159018706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:00:01.157661409 +0000 UTC m=+2036.804444815" watchObservedRunningTime="2026-07-30 03:00:01.159018706 +0000 UTC m=+2036.805802111"
Jul 30 03:00:02.138468 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:02.138431    2565 generic.go:356] "Generic (PLEG): container finished" podID="6a8e67e6-6266-4420-82b7-4f70e0678a9f" containerID="fa32286423813dc4fe99dfcc5906e3115a255155dc6bcaca9d97787f618d1f5b" exitCode=0
Jul 30 03:00:02.138918 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:02.138525    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756340-42grx" event={"ID":"6a8e67e6-6266-4420-82b7-4f70e0678a9f","Type":"ContainerDied","Data":"fa32286423813dc4fe99dfcc5906e3115a255155dc6bcaca9d97787f618d1f5b"}
Jul 30 03:00:02.139971 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:02.139952    2565 generic.go:356] "Generic (PLEG): container finished" podID="24701fca-d6a5-4c4e-8126-cd0c773b7aef" containerID="232c140f8ffc80e492a6eb7106e758859e4c93718b5a812451a0a616b2a8188f" exitCode=0
Jul 30 03:00:02.140036 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:02.139980    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1ca4664c-29756340-sfdg8" event={"ID":"24701fca-d6a5-4c4e-8126-cd0c773b7aef","Type":"ContainerDied","Data":"232c140f8ffc80e492a6eb7106e758859e4c93718b5a812451a0a616b2a8188f"}
Jul 30 03:00:03.153305 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:03.153272    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ca4664c-jrm9s"]
Jul 30 03:00:03.155859 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:03.155837    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ca4664c-jrm9s"
Jul 30 03:00:03.167845 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:03.167820    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ca4664c-jrm9s"]
Jul 30 03:00:03.257730 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:03.257699    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c94f3b5-4d24-41fd-8778-99dff3e1b24f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1ca4664c-jrm9s\" (UID: \"0c94f3b5-4d24-41fd-8778-99dff3e1b24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ca4664c-jrm9s"
Jul 30 03:00:03.303035 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:03.303010    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756340-42grx"
Jul 30 03:00:03.330214 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:03.330193    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1ca4664c-29756340-sfdg8"
Jul 30 03:00:03.358855 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:03.358826    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c94f3b5-4d24-41fd-8778-99dff3e1b24f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1ca4664c-jrm9s\" (UID: \"0c94f3b5-4d24-41fd-8778-99dff3e1b24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ca4664c-jrm9s"
Jul 30 03:00:03.359471 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:03.359445    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c94f3b5-4d24-41fd-8778-99dff3e1b24f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1ca4664c-jrm9s\" (UID: \"0c94f3b5-4d24-41fd-8778-99dff3e1b24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ca4664c-jrm9s"
Jul 30 03:00:03.469215 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:03.469103    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ca4664c-jrm9s"
Jul 30 03:00:03.606758 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:03.604648    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ca4664c-jrm9s"]
Jul 30 03:00:03.606758 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:00:03.606332    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c94f3b5_4d24_41fd_8778_99dff3e1b24f.slice/crio-8c4f6538a89b5b912670efc26886ed7487ea6f21cb4a4767eea3370a6da4b584 WatchSource:0}: Error finding container 8c4f6538a89b5b912670efc26886ed7487ea6f21cb4a4767eea3370a6da4b584: Status 404 returned error can't find the container with id 8c4f6538a89b5b912670efc26886ed7487ea6f21cb4a4767eea3370a6da4b584
Jul 30 03:00:04.154379 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:04.154346    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756340-42grx"
Jul 30 03:00:04.154379 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:04.154357    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756340-42grx" event={"ID":"6a8e67e6-6266-4420-82b7-4f70e0678a9f","Type":"ContainerDied","Data":"f89294ef4f1b091c5920d119f287a960f1a4d6a7d8c050c3eff5dff4874985e9"}
Jul 30 03:00:04.154813 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:04.154425    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f89294ef4f1b091c5920d119f287a960f1a4d6a7d8c050c3eff5dff4874985e9"
Jul 30 03:00:04.155778 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:04.155759    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1ca4664c-29756340-sfdg8"
Jul 30 03:00:04.155917 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:04.155754    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1ca4664c-29756340-sfdg8" event={"ID":"24701fca-d6a5-4c4e-8126-cd0c773b7aef","Type":"ContainerDied","Data":"2c57db896d574bbffec7e96ee6ed3508bd49f01c89bc2604cbe7229b14f4ae1f"}
Jul 30 03:00:04.155917 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:04.155903    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c57db896d574bbffec7e96ee6ed3508bd49f01c89bc2604cbe7229b14f4ae1f"
Jul 30 03:00:04.157716 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:04.157692    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ca4664c-jrm9s" event={"ID":"0c94f3b5-4d24-41fd-8778-99dff3e1b24f","Type":"ContainerStarted","Data":"0fd3d9d41f57aba357a4d18f25ac623498f73db9df121a9591176876c5f3a65f"}
Jul 30 03:00:04.157822 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:04.157737    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ca4664c-jrm9s" event={"ID":"0c94f3b5-4d24-41fd-8778-99dff3e1b24f","Type":"ContainerStarted","Data":"8c4f6538a89b5b912670efc26886ed7487ea6f21cb4a4767eea3370a6da4b584"}
Jul 30 03:00:04.173872 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:04.173828    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ca4664c-jrm9s" podStartSLOduration=1.17381398 podStartE2EDuration="1.17381398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:00:04.172099873 +0000 UTC m=+2039.818883279" watchObservedRunningTime="2026-07-30 03:00:04.17381398 +0000 UTC m=+2039.820597386"
Jul 30 03:00:05.165451 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:05.165415    2565 generic.go:356] "Generic (PLEG): container finished" podID="0c94f3b5-4d24-41fd-8778-99dff3e1b24f" containerID="0fd3d9d41f57aba357a4d18f25ac623498f73db9df121a9591176876c5f3a65f" exitCode=0
Jul 30 03:00:05.165860 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:05.165508    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ca4664c-jrm9s" event={"ID":"0c94f3b5-4d24-41fd-8778-99dff3e1b24f","Type":"ContainerDied","Data":"0fd3d9d41f57aba357a4d18f25ac623498f73db9df121a9591176876c5f3a65f"}
Jul 30 03:00:06.308815 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:06.308782    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ca4664c-jrm9s"
Jul 30 03:00:06.386711 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:06.386680    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c94f3b5-4d24-41fd-8778-99dff3e1b24f-maas-api-service-ca\") pod \"0c94f3b5-4d24-41fd-8778-99dff3e1b24f\" (UID: \"0c94f3b5-4d24-41fd-8778-99dff3e1b24f\") "
Jul 30 03:00:06.387067 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:06.387042    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c94f3b5-4d24-41fd-8778-99dff3e1b24f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0c94f3b5-4d24-41fd-8778-99dff3e1b24f" (UID: "0c94f3b5-4d24-41fd-8778-99dff3e1b24f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:00:06.488339 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:06.488257    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c94f3b5-4d24-41fd-8778-99dff3e1b24f-maas-api-service-ca\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:00:07.179805 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:07.179717    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ca4664c-jrm9s" event={"ID":"0c94f3b5-4d24-41fd-8778-99dff3e1b24f","Type":"ContainerDied","Data":"8c4f6538a89b5b912670efc26886ed7487ea6f21cb4a4767eea3370a6da4b584"}
Jul 30 03:00:07.179805 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:07.179764    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c4f6538a89b5b912670efc26886ed7487ea6f21cb4a4767eea3370a6da4b584"
Jul 30 03:00:07.179805 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:07.179727    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ca4664c-jrm9s"
Jul 30 03:00:13.572157 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:13.572120    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"]
Jul 30 03:00:13.572631 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:13.572400    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm" podUID="52713660-da7d-408b-832e-d33f9e298e10" containerName="maas-api" containerID="cri-o://f79754e2bc3bc8ad8d947a8c7cefcf97bdc3802d2871fa622a453c386e1c188e" gracePeriod=30
Jul 30 03:00:13.597209 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:13.597180    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1ca4664c-29756340-sfdg8"]
Jul 30 03:00:13.602135 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:13.602105    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-1ca4664c-29756340-sfdg8"]
Jul 30 03:00:13.627123 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:13.627091    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd"]
Jul 30 03:00:13.627381 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:13.627354    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd" podUID="6ea31c03-28fa-43e9-af77-5539bef9e23c" containerName="payload-processing" containerID="cri-o://d74332cab9360d23915039db5a64f0f28d6121ed66cc86e9ed26a7f7bb10de37" gracePeriod=30
Jul 30 03:00:13.893781 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:13.893754    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"
Jul 30 03:00:13.915264 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:13.915240    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd"
Jul 30 03:00:14.071796 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.071758    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzldk\" (UniqueName: \"kubernetes.io/projected/52713660-da7d-408b-832e-d33f9e298e10-kube-api-access-gzldk\") pod \"52713660-da7d-408b-832e-d33f9e298e10\" (UID: \"52713660-da7d-408b-832e-d33f9e298e10\") "
Jul 30 03:00:14.071980 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.071824    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ea31c03-28fa-43e9-af77-5539bef9e23c-plugins-config-volume\") pod \"6ea31c03-28fa-43e9-af77-5539bef9e23c\" (UID: \"6ea31c03-28fa-43e9-af77-5539bef9e23c\") "
Jul 30 03:00:14.071980 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.071939    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52713660-da7d-408b-832e-d33f9e298e10-openshift-service-ca\") pod \"52713660-da7d-408b-832e-d33f9e298e10\" (UID: \"52713660-da7d-408b-832e-d33f9e298e10\") "
Jul 30 03:00:14.072107 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.072034    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52713660-da7d-408b-832e-d33f9e298e10-maas-api-tls\") pod \"52713660-da7d-408b-832e-d33f9e298e10\" (UID: \"52713660-da7d-408b-832e-d33f9e298e10\") "
Jul 30 03:00:14.072107 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.072071    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq5fj\" (UniqueName: \"kubernetes.io/projected/6ea31c03-28fa-43e9-af77-5539bef9e23c-kube-api-access-tq5fj\") pod \"6ea31c03-28fa-43e9-af77-5539bef9e23c\" (UID: \"6ea31c03-28fa-43e9-af77-5539bef9e23c\") "
Jul 30 03:00:14.072371 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.072342    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea31c03-28fa-43e9-af77-5539bef9e23c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6ea31c03-28fa-43e9-af77-5539bef9e23c" (UID: "6ea31c03-28fa-43e9-af77-5539bef9e23c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:00:14.072541 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.072442    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52713660-da7d-408b-832e-d33f9e298e10-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "52713660-da7d-408b-832e-d33f9e298e10" (UID: "52713660-da7d-408b-832e-d33f9e298e10"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:00:14.074536 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.074515    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52713660-da7d-408b-832e-d33f9e298e10-kube-api-access-gzldk" (OuterVolumeSpecName: "kube-api-access-gzldk") pod "52713660-da7d-408b-832e-d33f9e298e10" (UID: "52713660-da7d-408b-832e-d33f9e298e10"). InnerVolumeSpecName "kube-api-access-gzldk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:00:14.074726 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.074708    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea31c03-28fa-43e9-af77-5539bef9e23c-kube-api-access-tq5fj" (OuterVolumeSpecName: "kube-api-access-tq5fj") pod "6ea31c03-28fa-43e9-af77-5539bef9e23c" (UID: "6ea31c03-28fa-43e9-af77-5539bef9e23c"). InnerVolumeSpecName "kube-api-access-tq5fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:00:14.074818 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.074798    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52713660-da7d-408b-832e-d33f9e298e10-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "52713660-da7d-408b-832e-d33f9e298e10" (UID: "52713660-da7d-408b-832e-d33f9e298e10"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:00:14.173362 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.173274    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52713660-da7d-408b-832e-d33f9e298e10-maas-api-tls\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:00:14.173362 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.173316    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq5fj\" (UniqueName: \"kubernetes.io/projected/6ea31c03-28fa-43e9-af77-5539bef9e23c-kube-api-access-tq5fj\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:00:14.173362 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.173334    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzldk\" (UniqueName: \"kubernetes.io/projected/52713660-da7d-408b-832e-d33f9e298e10-kube-api-access-gzldk\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:00:14.173362 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.173347    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ea31c03-28fa-43e9-af77-5539bef9e23c-plugins-config-volume\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:00:14.173362 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.173361    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52713660-da7d-408b-832e-d33f9e298e10-openshift-service-ca\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:00:14.220840 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.220786    2565 generic.go:356] "Generic (PLEG): container finished" podID="6ea31c03-28fa-43e9-af77-5539bef9e23c" containerID="d74332cab9360d23915039db5a64f0f28d6121ed66cc86e9ed26a7f7bb10de37" exitCode=0
Jul 30 03:00:14.221021 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.220851    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd"
Jul 30 03:00:14.221021 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.220872    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd" event={"ID":"6ea31c03-28fa-43e9-af77-5539bef9e23c","Type":"ContainerDied","Data":"d74332cab9360d23915039db5a64f0f28d6121ed66cc86e9ed26a7f7bb10de37"}
Jul 30 03:00:14.221021 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.220947    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd" event={"ID":"6ea31c03-28fa-43e9-af77-5539bef9e23c","Type":"ContainerDied","Data":"995b795ef0c706be9c963e03dbc608649fe3ed80591de5a674192eb09c3e45ac"}
Jul 30 03:00:14.221021 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.220972    2565 scope.go:117] "RemoveContainer" containerID="d74332cab9360d23915039db5a64f0f28d6121ed66cc86e9ed26a7f7bb10de37"
Jul 30 03:00:14.222234 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.222214    2565 generic.go:356] "Generic (PLEG): container finished" podID="52713660-da7d-408b-832e-d33f9e298e10" containerID="f79754e2bc3bc8ad8d947a8c7cefcf97bdc3802d2871fa622a453c386e1c188e" exitCode=0
Jul 30 03:00:14.222315 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.222279    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"
Jul 30 03:00:14.222315 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.222295    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm" event={"ID":"52713660-da7d-408b-832e-d33f9e298e10","Type":"ContainerDied","Data":"f79754e2bc3bc8ad8d947a8c7cefcf97bdc3802d2871fa622a453c386e1c188e"}
Jul 30 03:00:14.222392 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.222351    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm" event={"ID":"52713660-da7d-408b-832e-d33f9e298e10","Type":"ContainerDied","Data":"0e1fdc5ae0bd7c813743d8ac52fceef72286bcd60e6ac70e028d1ebf9448be33"}
Jul 30 03:00:14.231376 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.231355    2565 scope.go:117] "RemoveContainer" containerID="d74332cab9360d23915039db5a64f0f28d6121ed66cc86e9ed26a7f7bb10de37"
Jul 30 03:00:14.231654 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:00:14.231628    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d74332cab9360d23915039db5a64f0f28d6121ed66cc86e9ed26a7f7bb10de37\": container with ID starting with d74332cab9360d23915039db5a64f0f28d6121ed66cc86e9ed26a7f7bb10de37 not found: ID does not exist" containerID="d74332cab9360d23915039db5a64f0f28d6121ed66cc86e9ed26a7f7bb10de37"
Jul 30 03:00:14.231748 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.231663    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d74332cab9360d23915039db5a64f0f28d6121ed66cc86e9ed26a7f7bb10de37"} err="failed to get container status \"d74332cab9360d23915039db5a64f0f28d6121ed66cc86e9ed26a7f7bb10de37\": rpc error: code = NotFound desc = could not find container \"d74332cab9360d23915039db5a64f0f28d6121ed66cc86e9ed26a7f7bb10de37\": container with ID starting with d74332cab9360d23915039db5a64f0f28d6121ed66cc86e9ed26a7f7bb10de37 not found: ID does not exist"
Jul 30 03:00:14.231748 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.231682    2565 scope.go:117] "RemoveContainer" containerID="f79754e2bc3bc8ad8d947a8c7cefcf97bdc3802d2871fa622a453c386e1c188e"
Jul 30 03:00:14.240041 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.240021    2565 scope.go:117] "RemoveContainer" containerID="f79754e2bc3bc8ad8d947a8c7cefcf97bdc3802d2871fa622a453c386e1c188e"
Jul 30 03:00:14.240316 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:00:14.240295    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f79754e2bc3bc8ad8d947a8c7cefcf97bdc3802d2871fa622a453c386e1c188e\": container with ID starting with f79754e2bc3bc8ad8d947a8c7cefcf97bdc3802d2871fa622a453c386e1c188e not found: ID does not exist" containerID="f79754e2bc3bc8ad8d947a8c7cefcf97bdc3802d2871fa622a453c386e1c188e"
Jul 30 03:00:14.240383 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.240323    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f79754e2bc3bc8ad8d947a8c7cefcf97bdc3802d2871fa622a453c386e1c188e"} err="failed to get container status \"f79754e2bc3bc8ad8d947a8c7cefcf97bdc3802d2871fa622a453c386e1c188e\": rpc error: code = NotFound desc = could not find container \"f79754e2bc3bc8ad8d947a8c7cefcf97bdc3802d2871fa622a453c386e1c188e\": container with ID starting with f79754e2bc3bc8ad8d947a8c7cefcf97bdc3802d2871fa622a453c386e1c188e not found: ID does not exist"
Jul 30 03:00:14.245956 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.245936    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd"]
Jul 30 03:00:14.251680 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.251659    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1ca4664c-58b9f4fbbb-969fd"]
Jul 30 03:00:14.262090 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.262065    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"]
Jul 30 03:00:14.266118 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.266098    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1ca4664c-79df65d49-nblvm"]
Jul 30 03:00:14.800969 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.800937    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24701fca-d6a5-4c4e-8126-cd0c773b7aef" path="/var/lib/kubelet/pods/24701fca-d6a5-4c4e-8126-cd0c773b7aef/volumes"
Jul 30 03:00:14.801365 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.801326    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52713660-da7d-408b-832e-d33f9e298e10" path="/var/lib/kubelet/pods/52713660-da7d-408b-832e-d33f9e298e10/volumes"
Jul 30 03:00:14.801634 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:14.801622    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ea31c03-28fa-43e9-af77-5539bef9e23c" path="/var/lib/kubelet/pods/6ea31c03-28fa-43e9-af77-5539bef9e23c/volumes"
Jul 30 03:00:18.258960 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:18.258928    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ca4664c-jrm9s"]
Jul 30 03:00:18.262302 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:18.262278    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ca4664c-jrm9s"]
Jul 30 03:00:18.420269 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:18.420238    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"]
Jul 30 03:00:18.420505 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:18.420481    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx" podUID="ccd3f4fa-3ae5-4991-9523-950cb412bfc9" containerName="istio-proxy" containerID="cri-o://52b9f176cd83345c955740113bffffbb41eb3c48aadd52be518bc3d01f782ee1" gracePeriod=30
Jul 30 03:00:18.801458 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:18.801424    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c94f3b5-4d24-41fd-8778-99dff3e1b24f" path="/var/lib/kubelet/pods/0c94f3b5-4d24-41fd-8778-99dff3e1b24f/volumes"
Jul 30 03:00:23.688595 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.688527    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 03:00:23.759433 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.759391    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npt4w\" (UniqueName: \"kubernetes.io/projected/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-kube-api-access-npt4w\") pod \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") "
Jul 30 03:00:23.759433 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.759434    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-workload-socket\") pod \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") "
Jul 30 03:00:23.759682 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.759457    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istiod-ca-cert\") pod \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") "
Jul 30 03:00:23.759682 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.759478    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-credential-socket\") pod \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") "
Jul 30 03:00:23.759682 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.759509    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-envoy\") pod \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") "
Jul 30 03:00:23.759682 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.759552    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-podinfo\") pod \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") "
Jul 30 03:00:23.759682 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.759584    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-data\") pod \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") "
Jul 30 03:00:23.759682 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.759631    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-token\") pod \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") "
Jul 30 03:00:23.759998 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.759707    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-workload-certs\") pod \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\" (UID: \"ccd3f4fa-3ae5-4991-9523-950cb412bfc9\") "
Jul 30 03:00:23.759998 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.759751    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ccd3f4fa-3ae5-4991-9523-950cb412bfc9" (UID: "ccd3f4fa-3ae5-4991-9523-950cb412bfc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:00:23.759998 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.759842    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ccd3f4fa-3ae5-4991-9523-950cb412bfc9" (UID: "ccd3f4fa-3ae5-4991-9523-950cb412bfc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:00:23.760151 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.760012    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ccd3f4fa-3ae5-4991-9523-950cb412bfc9" (UID: "ccd3f4fa-3ae5-4991-9523-950cb412bfc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:00:23.760151 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.760061    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ccd3f4fa-3ae5-4991-9523-950cb412bfc9" (UID: "ccd3f4fa-3ae5-4991-9523-950cb412bfc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:00:23.760151 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.760089    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-workload-certs\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:00:23.760151 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.760106    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-workload-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:00:23.760151 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.760121    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istiod-ca-cert\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:00:23.760915 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.760889    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-data" (OuterVolumeSpecName: "istio-data") pod "ccd3f4fa-3ae5-4991-9523-950cb412bfc9" (UID: "ccd3f4fa-3ae5-4991-9523-950cb412bfc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:00:23.761957 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.761934    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ccd3f4fa-3ae5-4991-9523-950cb412bfc9" (UID: "ccd3f4fa-3ae5-4991-9523-950cb412bfc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:00:23.762064 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.761966    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-token" (OuterVolumeSpecName: "istio-token") pod "ccd3f4fa-3ae5-4991-9523-950cb412bfc9" (UID: "ccd3f4fa-3ae5-4991-9523-950cb412bfc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:00:23.762259 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.762241    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-kube-api-access-npt4w" (OuterVolumeSpecName: "kube-api-access-npt4w") pod "ccd3f4fa-3ae5-4991-9523-950cb412bfc9" (UID: "ccd3f4fa-3ae5-4991-9523-950cb412bfc9"). InnerVolumeSpecName "kube-api-access-npt4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:00:23.762259 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.762245    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ccd3f4fa-3ae5-4991-9523-950cb412bfc9" (UID: "ccd3f4fa-3ae5-4991-9523-950cb412bfc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 30 03:00:23.861185 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.861123    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npt4w\" (UniqueName: \"kubernetes.io/projected/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-kube-api-access-npt4w\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:00:23.861348 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.861284    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-credential-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:00:23.861348 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.861309    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-envoy\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:00:23.861348 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.861319    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-podinfo\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:00:23.861348 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.861327    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-data\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:00:23.861348 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:23.861337    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd3f4fa-3ae5-4991-9523-950cb412bfc9-istio-token\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:00:24.284368 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:24.284337    2565 generic.go:356] "Generic (PLEG): container finished" podID="ccd3f4fa-3ae5-4991-9523-950cb412bfc9" containerID="52b9f176cd83345c955740113bffffbb41eb3c48aadd52be518bc3d01f782ee1" exitCode=0
Jul 30 03:00:24.284551 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:24.284374    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx" event={"ID":"ccd3f4fa-3ae5-4991-9523-950cb412bfc9","Type":"ContainerDied","Data":"52b9f176cd83345c955740113bffffbb41eb3c48aadd52be518bc3d01f782ee1"}
Jul 30 03:00:24.284551 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:24.284384    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"
Jul 30 03:00:24.284551 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:24.284417    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx" event={"ID":"ccd3f4fa-3ae5-4991-9523-950cb412bfc9","Type":"ContainerDied","Data":"6cdb09cf3e0c3efc049b5c9805aeced0bbd659b1f5a6ddb6fb5f65f86a197dbf"}
Jul 30 03:00:24.284551 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:24.284440    2565 scope.go:117] "RemoveContainer" containerID="52b9f176cd83345c955740113bffffbb41eb3c48aadd52be518bc3d01f782ee1"
Jul 30 03:00:24.294590 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:24.294571    2565 scope.go:117] "RemoveContainer" containerID="cad5263dd86af6480d422245c6cc08a57d0f4261935297867aae4ec2ea4d7a01"
Jul 30 03:00:24.302910 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:24.302891    2565 scope.go:117] "RemoveContainer" containerID="52b9f176cd83345c955740113bffffbb41eb3c48aadd52be518bc3d01f782ee1"
Jul 30 03:00:24.303153 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:00:24.303136    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52b9f176cd83345c955740113bffffbb41eb3c48aadd52be518bc3d01f782ee1\": container with ID starting with 52b9f176cd83345c955740113bffffbb41eb3c48aadd52be518bc3d01f782ee1 not found: ID does not exist" containerID="52b9f176cd83345c955740113bffffbb41eb3c48aadd52be518bc3d01f782ee1"
Jul 30 03:00:24.303236 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:24.303161    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b9f176cd83345c955740113bffffbb41eb3c48aadd52be518bc3d01f782ee1"} err="failed to get container status \"52b9f176cd83345c955740113bffffbb41eb3c48aadd52be518bc3d01f782ee1\": rpc error: code = NotFound desc = could not find container \"52b9f176cd83345c955740113bffffbb41eb3c48aadd52be518bc3d01f782ee1\": container with ID starting with 52b9f176cd83345c955740113bffffbb41eb3c48aadd52be518bc3d01f782ee1 not found: ID does not exist"
Jul 30 03:00:24.303236 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:24.303201    2565 scope.go:117] "RemoveContainer" containerID="cad5263dd86af6480d422245c6cc08a57d0f4261935297867aae4ec2ea4d7a01"
Jul 30 03:00:24.303467 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:00:24.303447    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad5263dd86af6480d422245c6cc08a57d0f4261935297867aae4ec2ea4d7a01\": container with ID starting with cad5263dd86af6480d422245c6cc08a57d0f4261935297867aae4ec2ea4d7a01 not found: ID does not exist" containerID="cad5263dd86af6480d422245c6cc08a57d0f4261935297867aae4ec2ea4d7a01"
Jul 30 03:00:24.303511 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:24.303472    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad5263dd86af6480d422245c6cc08a57d0f4261935297867aae4ec2ea4d7a01"} err="failed to get container status \"cad5263dd86af6480d422245c6cc08a57d0f4261935297867aae4ec2ea4d7a01\": rpc error: code = NotFound desc = could not find container \"cad5263dd86af6480d422245c6cc08a57d0f4261935297867aae4ec2ea4d7a01\": container with ID starting with cad5263dd86af6480d422245c6cc08a57d0f4261935297867aae4ec2ea4d7a01 not found: ID does not exist"
Jul 30 03:00:24.311839 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:24.311817    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"]
Jul 30 03:00:24.314986 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:24.314966    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-1ca4664c-openshift-default-57d944b6b8-mnjgx"]
Jul 30 03:00:24.801613 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:00:24.801579    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccd3f4fa-3ae5-4991-9523-950cb412bfc9" path="/var/lib/kubelet/pods/ccd3f4fa-3ae5-4991-9523-950cb412bfc9/volumes"
Jul 30 03:01:54.806269 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806232    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"]
Jul 30 03:01:54.806759 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806627    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea31c03-28fa-43e9-af77-5539bef9e23c" containerName="payload-processing"
Jul 30 03:01:54.806759 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806639    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea31c03-28fa-43e9-af77-5539bef9e23c" containerName="payload-processing"
Jul 30 03:01:54.806759 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806651    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24701fca-d6a5-4c4e-8126-cd0c773b7aef" containerName="cleanup"
Jul 30 03:01:54.806759 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806656    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="24701fca-d6a5-4c4e-8126-cd0c773b7aef" containerName="cleanup"
Jul 30 03:01:54.806759 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806663    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c94f3b5-4d24-41fd-8778-99dff3e1b24f" containerName="revoke-keys"
Jul 30 03:01:54.806759 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806670    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c94f3b5-4d24-41fd-8778-99dff3e1b24f" containerName="revoke-keys"
Jul 30 03:01:54.806759 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806677    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a8e67e6-6266-4420-82b7-4f70e0678a9f" containerName="cleanup"
Jul 30 03:01:54.806759 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806682    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a8e67e6-6266-4420-82b7-4f70e0678a9f" containerName="cleanup"
Jul 30 03:01:54.806759 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806699    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccd3f4fa-3ae5-4991-9523-950cb412bfc9" containerName="istio-proxy"
Jul 30 03:01:54.806759 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806704    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccd3f4fa-3ae5-4991-9523-950cb412bfc9" containerName="istio-proxy"
Jul 30 03:01:54.806759 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806711    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccd3f4fa-3ae5-4991-9523-950cb412bfc9" containerName="istio-proxy"
Jul 30 03:01:54.806759 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806716    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccd3f4fa-3ae5-4991-9523-950cb412bfc9" containerName="istio-proxy"
Jul 30 03:01:54.806759 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806724    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52713660-da7d-408b-832e-d33f9e298e10" containerName="maas-api"
Jul 30 03:01:54.806759 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806729    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="52713660-da7d-408b-832e-d33f9e298e10" containerName="maas-api"
Jul 30 03:01:54.807198 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806787    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c94f3b5-4d24-41fd-8778-99dff3e1b24f" containerName="revoke-keys"
Jul 30 03:01:54.807198 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806794    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccd3f4fa-3ae5-4991-9523-950cb412bfc9" containerName="istio-proxy"
Jul 30 03:01:54.807198 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806801    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccd3f4fa-3ae5-4991-9523-950cb412bfc9" containerName="istio-proxy"
Jul 30 03:01:54.807198 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806810    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a8e67e6-6266-4420-82b7-4f70e0678a9f" containerName="cleanup"
Jul 30 03:01:54.807198 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806816    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="24701fca-d6a5-4c4e-8126-cd0c773b7aef" containerName="cleanup"
Jul 30 03:01:54.807198 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806821    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="52713660-da7d-408b-832e-d33f9e298e10" containerName="maas-api"
Jul 30 03:01:54.807198 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.806826    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ea31c03-28fa-43e9-af77-5539bef9e23c" containerName="payload-processing"
Jul 30 03:01:54.809846 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.809826    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:54.813456 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.813436    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fad13d10-openshift-default-dockercfg-c86jv\""
Jul 30 03:01:54.845943 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.845906    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"]
Jul 30 03:01:54.968065 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.968030    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-envoy\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:54.968065 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.968065    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istiod-ca-cert\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:54.968306 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.968131    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-credential-socket\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:54.968306 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.968190    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-workload-socket\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:54.968306 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.968218    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-podinfo\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:54.968306 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.968236    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-data\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:54.968306 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.968301    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-workload-certs\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:54.968469 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.968357    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-token\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:54.968469 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:54.968377    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nwnw\" (UniqueName: \"kubernetes.io/projected/9e928e41-9555-47c4-beaa-ee4c874c0c0c-kube-api-access-7nwnw\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:55.069040 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.068955    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-token\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:55.069040 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.068991    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nwnw\" (UniqueName: \"kubernetes.io/projected/9e928e41-9555-47c4-beaa-ee4c874c0c0c-kube-api-access-7nwnw\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:55.069040 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.069023    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-envoy\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:55.069370 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.069134    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istiod-ca-cert\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:55.069370 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.069230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-credential-socket\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:55.069370 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.069268    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-workload-socket\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:55.069370 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.069295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-podinfo\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:55.069370 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.069318    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-data\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:55.069631 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.069384    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-workload-certs\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:55.069631 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.069551    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-credential-socket\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:55.069741 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.069656    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-workload-socket\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:55.069741 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.069661    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-data\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:55.069860 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.069737    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-workload-certs\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:55.069969 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.069949    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istiod-ca-cert\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:55.071616 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.071590    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-envoy\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:55.072320 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.072298    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-podinfo\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:55.078987 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.078957    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-token\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:55.079088 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.079020    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nwnw\" (UniqueName: \"kubernetes.io/projected/9e928e41-9555-47c4-beaa-ee4c874c0c0c-kube-api-access-7nwnw\") pod \"e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") " pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:55.119397 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.119352    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:55.269325 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.269296    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"]
Jul 30 03:01:55.271419 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:01:55.271386    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e928e41_9555_47c4_beaa_ee4c874c0c0c.slice/crio-0cc6324c86379b4c094145fb655cc078194aed7f030c598b81a56b3416c1d6a8 WatchSource:0}: Error finding container 0cc6324c86379b4c094145fb655cc078194aed7f030c598b81a56b3416c1d6a8: Status 404 returned error can't find the container with id 0cc6324c86379b4c094145fb655cc078194aed7f030c598b81a56b3416c1d6a8
Jul 30 03:01:55.273880 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.273836    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:01:55.273992 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.273927    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:01:55.274064 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.273977    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:01:55.809011 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.808978    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc" event={"ID":"9e928e41-9555-47c4-beaa-ee4c874c0c0c","Type":"ContainerStarted","Data":"d1137f59e86be0f4c6678f2180e7366f03b964d904ac484f70eb79117d163c8d"}
Jul 30 03:01:55.809443 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.809039    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc" event={"ID":"9e928e41-9555-47c4-beaa-ee4c874c0c0c","Type":"ContainerStarted","Data":"0cc6324c86379b4c094145fb655cc078194aed7f030c598b81a56b3416c1d6a8"}
Jul 30 03:01:55.831875 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:55.831826    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc" podStartSLOduration=1.8318096050000001 podStartE2EDuration="1.831809605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:01:55.82973976 +0000 UTC m=+2151.476523166" watchObservedRunningTime="2026-07-30 03:01:55.831809605 +0000 UTC m=+2151.478593010"
Jul 30 03:01:56.120232 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:56.120135    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:56.124840 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:56.124817    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:56.814711 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:56.814680    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:56.815744 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:56.815724    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:01:57.928317 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:57.928281    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"]
Jul 30 03:01:57.932309 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:57.932286    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"
Jul 30 03:01:57.935052 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:57.935029    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fad13d10-serving-cert\""
Jul 30 03:01:57.943349 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:57.943313    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"]
Jul 30 03:01:58.099706 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.099670    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6200de-b079-4f6b-935c-787acea31c82-maas-api-tls\") pod \"maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg\" (UID: \"fb6200de-b079-4f6b-935c-787acea31c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"
Jul 30 03:01:58.099907 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.099798    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6200de-b079-4f6b-935c-787acea31c82-openshift-service-ca\") pod \"maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg\" (UID: \"fb6200de-b079-4f6b-935c-787acea31c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"
Jul 30 03:01:58.099907 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.099823    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf6xk\" (UniqueName: \"kubernetes.io/projected/fb6200de-b079-4f6b-935c-787acea31c82-kube-api-access-tf6xk\") pod \"maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg\" (UID: \"fb6200de-b079-4f6b-935c-787acea31c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"
Jul 30 03:01:58.201102 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.201000    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6200de-b079-4f6b-935c-787acea31c82-openshift-service-ca\") pod \"maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg\" (UID: \"fb6200de-b079-4f6b-935c-787acea31c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"
Jul 30 03:01:58.201102 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.201041    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf6xk\" (UniqueName: \"kubernetes.io/projected/fb6200de-b079-4f6b-935c-787acea31c82-kube-api-access-tf6xk\") pod \"maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg\" (UID: \"fb6200de-b079-4f6b-935c-787acea31c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"
Jul 30 03:01:58.201102 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.201098    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6200de-b079-4f6b-935c-787acea31c82-maas-api-tls\") pod \"maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg\" (UID: \"fb6200de-b079-4f6b-935c-787acea31c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"
Jul 30 03:01:58.201784 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.201757    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6200de-b079-4f6b-935c-787acea31c82-openshift-service-ca\") pod \"maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg\" (UID: \"fb6200de-b079-4f6b-935c-787acea31c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"
Jul 30 03:01:58.203759 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.203736    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6200de-b079-4f6b-935c-787acea31c82-maas-api-tls\") pod \"maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg\" (UID: \"fb6200de-b079-4f6b-935c-787acea31c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"
Jul 30 03:01:58.209337 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.209308    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf6xk\" (UniqueName: \"kubernetes.io/projected/fb6200de-b079-4f6b-935c-787acea31c82-kube-api-access-tf6xk\") pod \"maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg\" (UID: \"fb6200de-b079-4f6b-935c-787acea31c82\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"
Jul 30 03:01:58.243960 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.243916    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"
Jul 30 03:01:58.403613 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.402777    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"]
Jul 30 03:01:58.405410 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:01:58.405382    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6200de_b079_4f6b_935c_787acea31c82.slice/crio-b269be84c9418a3aa6568e4cff18c08b7d5d35518392a67fe1a9e0e7085295ca WatchSource:0}: Error finding container b269be84c9418a3aa6568e4cff18c08b7d5d35518392a67fe1a9e0e7085295ca: Status 404 returned error can't find the container with id b269be84c9418a3aa6568e4cff18c08b7d5d35518392a67fe1a9e0e7085295ca
Jul 30 03:01:58.600865 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.600829    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29"]
Jul 30 03:01:58.605549 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.605529    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29"
Jul 30 03:01:58.611116 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.611093    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fad13d10-dockercfg-76skn\""
Jul 30 03:01:58.612003 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.611958    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fad13d10\""
Jul 30 03:01:58.615238 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.615201    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29"]
Jul 30 03:01:58.624873 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.624839    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp"]
Jul 30 03:01:58.628511 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.628489    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp"
Jul 30 03:01:58.643678 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.643647    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp"]
Jul 30 03:01:58.707604 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.707564    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wwjz\" (UniqueName: \"kubernetes.io/projected/7964f74f-2180-4f18-a8b1-13f2e6f30375-kube-api-access-4wwjz\") pod \"payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29\" (UID: \"7964f74f-2180-4f18-a8b1-13f2e6f30375\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29"
Jul 30 03:01:58.707794 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.707757    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7964f74f-2180-4f18-a8b1-13f2e6f30375-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29\" (UID: \"7964f74f-2180-4f18-a8b1-13f2e6f30375\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29"
Jul 30 03:01:58.809180 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.809131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7964f74f-2180-4f18-a8b1-13f2e6f30375-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29\" (UID: \"7964f74f-2180-4f18-a8b1-13f2e6f30375\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29"
Jul 30 03:01:58.809379 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.809207    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd844156-2889-4ea1-be8d-703a7bcc8001-plugins-config-volume\") pod \"payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp\" (UID: \"cd844156-2889-4ea1-be8d-703a7bcc8001\") " pod="openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp"
Jul 30 03:01:58.809379 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.809249    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wwjz\" (UniqueName: \"kubernetes.io/projected/7964f74f-2180-4f18-a8b1-13f2e6f30375-kube-api-access-4wwjz\") pod \"payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29\" (UID: \"7964f74f-2180-4f18-a8b1-13f2e6f30375\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29"
Jul 30 03:01:58.809506 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.809374    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lklzz\" (UniqueName: \"kubernetes.io/projected/cd844156-2889-4ea1-be8d-703a7bcc8001-kube-api-access-lklzz\") pod \"payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp\" (UID: \"cd844156-2889-4ea1-be8d-703a7bcc8001\") " pod="openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp"
Jul 30 03:01:58.809912 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.809888    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7964f74f-2180-4f18-a8b1-13f2e6f30375-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29\" (UID: \"7964f74f-2180-4f18-a8b1-13f2e6f30375\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29"
Jul 30 03:01:58.818134 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.818110    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wwjz\" (UniqueName: \"kubernetes.io/projected/7964f74f-2180-4f18-a8b1-13f2e6f30375-kube-api-access-4wwjz\") pod \"payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29\" (UID: \"7964f74f-2180-4f18-a8b1-13f2e6f30375\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29"
Jul 30 03:01:58.826424 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.826395    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg" event={"ID":"fb6200de-b079-4f6b-935c-787acea31c82","Type":"ContainerStarted","Data":"b269be84c9418a3aa6568e4cff18c08b7d5d35518392a67fe1a9e0e7085295ca"}
Jul 30 03:01:58.910414 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.910386    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd844156-2889-4ea1-be8d-703a7bcc8001-plugins-config-volume\") pod \"payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp\" (UID: \"cd844156-2889-4ea1-be8d-703a7bcc8001\") " pod="openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp"
Jul 30 03:01:58.910561 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.910473    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lklzz\" (UniqueName: \"kubernetes.io/projected/cd844156-2889-4ea1-be8d-703a7bcc8001-kube-api-access-lklzz\") pod \"payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp\" (UID: \"cd844156-2889-4ea1-be8d-703a7bcc8001\") " pod="openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp"
Jul 30 03:01:58.911208 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.911184    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd844156-2889-4ea1-be8d-703a7bcc8001-plugins-config-volume\") pod \"payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp\" (UID: \"cd844156-2889-4ea1-be8d-703a7bcc8001\") " pod="openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp"
Jul 30 03:01:58.918116 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.918086    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29"
Jul 30 03:01:58.919142 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.919119    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lklzz\" (UniqueName: \"kubernetes.io/projected/cd844156-2889-4ea1-be8d-703a7bcc8001-kube-api-access-lklzz\") pod \"payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp\" (UID: \"cd844156-2889-4ea1-be8d-703a7bcc8001\") " pod="openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp"
Jul 30 03:01:58.939047 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:58.939005    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp"
Jul 30 03:01:59.079036 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:59.078981    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29"]
Jul 30 03:01:59.081351 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:01:59.081320    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7964f74f_2180_4f18_a8b1_13f2e6f30375.slice/crio-219f0efc0aaba3a82a668ff442b259ac4482275ecd2f288166d1e60fb7cfaaef WatchSource:0}: Error finding container 219f0efc0aaba3a82a668ff442b259ac4482275ecd2f288166d1e60fb7cfaaef: Status 404 returned error can't find the container with id 219f0efc0aaba3a82a668ff442b259ac4482275ecd2f288166d1e60fb7cfaaef
Jul 30 03:01:59.113135 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:59.113109    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp"]
Jul 30 03:01:59.117325 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:01:59.117301    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd844156_2889_4ea1_be8d_703a7bcc8001.slice/crio-f8de98dca7fd44c49bcafbe99ce344b43cede7f259307b95d169f98f6e845dbe WatchSource:0}: Error finding container f8de98dca7fd44c49bcafbe99ce344b43cede7f259307b95d169f98f6e845dbe: Status 404 returned error can't find the container with id f8de98dca7fd44c49bcafbe99ce344b43cede7f259307b95d169f98f6e845dbe
Jul 30 03:01:59.835463 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:59.835423    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp" event={"ID":"cd844156-2889-4ea1-be8d-703a7bcc8001","Type":"ContainerStarted","Data":"46f71fd922adb5d374a23fa14c3cc05a62772c3ed44ff04e4d652f6bddc3d0c9"}
Jul 30 03:01:59.835654 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:59.835487    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp"
Jul 30 03:01:59.835654 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:59.835508    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp" event={"ID":"cd844156-2889-4ea1-be8d-703a7bcc8001","Type":"ContainerStarted","Data":"f8de98dca7fd44c49bcafbe99ce344b43cede7f259307b95d169f98f6e845dbe"}
Jul 30 03:01:59.837086 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:59.837052    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29" event={"ID":"7964f74f-2180-4f18-a8b1-13f2e6f30375","Type":"ContainerStarted","Data":"cc437424a224b87ae7f8399cc923c4fba16b583852a5aeb93b1d7c57c13e5e97"}
Jul 30 03:01:59.837232 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:59.837117    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29"
Jul 30 03:01:59.837232 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:59.837141    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29" event={"ID":"7964f74f-2180-4f18-a8b1-13f2e6f30375","Type":"ContainerStarted","Data":"219f0efc0aaba3a82a668ff442b259ac4482275ecd2f288166d1e60fb7cfaaef"}
Jul 30 03:01:59.838469 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:59.838445    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg" event={"ID":"fb6200de-b079-4f6b-935c-787acea31c82","Type":"ContainerStarted","Data":"6bef8a8368902b348e036b50a19c7f6cc7052b33db8ce8dd1a82ae9661fb1b50"}
Jul 30 03:01:59.838569 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:59.838532    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"
Jul 30 03:01:59.860256 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:59.860206    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp" podStartSLOduration=1.86017805 podStartE2EDuration="1.86017805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:01:59.858093135 +0000 UTC m=+2155.504876541" watchObservedRunningTime="2026-07-30 03:01:59.86017805 +0000 UTC m=+2155.506961447"
Jul 30 03:01:59.874100 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:59.874032    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg" podStartSLOduration=2.431002502 podStartE2EDuration="2.874010449s" totalImagesPullingTime="443.007947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:01:59.873213241 +0000 UTC m=+2155.519996648" watchObservedRunningTime="2026-07-30 03:01:59.874010449 +0000 UTC m=+2155.520793855"
Jul 30 03:01:59.892108 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:01:59.892054    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29" podStartSLOduration=1.892038303 podStartE2EDuration="1.892038303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:01:59.890659295 +0000 UTC m=+2155.537442702" watchObservedRunningTime="2026-07-30 03:01:59.892038303 +0000 UTC m=+2155.538821769"
Jul 30 03:02:05.850641 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:05.850610    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"
Jul 30 03:02:08.894481 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:08.894444    2565 generic.go:356] "Generic (PLEG): container finished" podID="9e928e41-9555-47c4-beaa-ee4c874c0c0c" containerID="d1137f59e86be0f4c6678f2180e7366f03b964d904ac484f70eb79117d163c8d" exitCode=0
Jul 30 03:02:08.895005 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:08.894491    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc" event={"ID":"9e928e41-9555-47c4-beaa-ee4c874c0c0c","Type":"ContainerDied","Data":"d1137f59e86be0f4c6678f2180e7366f03b964d904ac484f70eb79117d163c8d"}
Jul 30 03:02:08.895005 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:08.894961    2565 scope.go:117] "RemoveContainer" containerID="d1137f59e86be0f4c6678f2180e7366f03b964d904ac484f70eb79117d163c8d"
Jul 30 03:02:08.896060 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:08.896024    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:02:08.896195 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:08.896114    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:02:08.896195 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:08.896161    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:02:09.522230 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:09.522191    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fad13d10-9n4bk"]
Jul 30 03:02:09.525779 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:09.525762    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fad13d10-9n4bk"
Jul 30 03:02:09.528658 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:09.528639    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2npk7\""
Jul 30 03:02:09.535801 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:09.535777    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fad13d10-9n4bk"]
Jul 30 03:02:09.609718 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:09.609685    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48569794-7865-465c-8b15-a8e7a21b55fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fad13d10-9n4bk\" (UID: \"48569794-7865-465c-8b15-a8e7a21b55fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fad13d10-9n4bk"
Jul 30 03:02:09.710678 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:09.710640    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48569794-7865-465c-8b15-a8e7a21b55fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fad13d10-9n4bk\" (UID: \"48569794-7865-465c-8b15-a8e7a21b55fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fad13d10-9n4bk"
Jul 30 03:02:09.711325 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:09.711302    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48569794-7865-465c-8b15-a8e7a21b55fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fad13d10-9n4bk\" (UID: \"48569794-7865-465c-8b15-a8e7a21b55fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fad13d10-9n4bk"
Jul 30 03:02:09.837072 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:09.837036    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fad13d10-9n4bk"
Jul 30 03:02:09.908341 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:09.908293    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc" event={"ID":"9e928e41-9555-47c4-beaa-ee4c874c0c0c","Type":"ContainerStarted","Data":"dfc2295f10c3eca375a1d61004b2524ef79ac25076244f3bb6d56dc304973228"}
Jul 30 03:02:10.032029 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:10.032006    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fad13d10-9n4bk"]
Jul 30 03:02:10.033272 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:02:10.033246    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48569794_7865_465c_8b15_a8e7a21b55fe.slice/crio-d40e0fd95f2bb3def0b491e0113d34be97cb06f9870b5a777138a7a9f3526544 WatchSource:0}: Error finding container d40e0fd95f2bb3def0b491e0113d34be97cb06f9870b5a777138a7a9f3526544: Status 404 returned error can't find the container with id d40e0fd95f2bb3def0b491e0113d34be97cb06f9870b5a777138a7a9f3526544
Jul 30 03:02:10.119780 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:10.119682    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:02:10.121118 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:10.121083    2565 patch_prober.go:28] interesting pod/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Jul 30 03:02:10.121286 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:10.121142    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc" podUID="9e928e41-9555-47c4-beaa-ee4c874c0c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Jul 30 03:02:10.847003 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:10.846973    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp"
Jul 30 03:02:10.847218 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:10.847114    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29"
Jul 30 03:02:10.921039 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:10.920987    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fad13d10-9n4bk" event={"ID":"48569794-7865-465c-8b15-a8e7a21b55fe","Type":"ContainerStarted","Data":"1a420fa0711c315ceaf26c12776abe7aa2f38174e376efb2e5ec3292e281ef3f"}
Jul 30 03:02:10.921451 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:10.921073    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fad13d10-9n4bk" event={"ID":"48569794-7865-465c-8b15-a8e7a21b55fe","Type":"ContainerStarted","Data":"d40e0fd95f2bb3def0b491e0113d34be97cb06f9870b5a777138a7a9f3526544"}
Jul 30 03:02:10.939771 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:10.939723    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fad13d10-9n4bk" podStartSLOduration=1.9397045469999998 podStartE2EDuration="1.939704547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:02:10.936499518 +0000 UTC m=+2166.583282923" watchObservedRunningTime="2026-07-30 03:02:10.939704547 +0000 UTC m=+2166.586487953"
Jul 30 03:02:11.120834 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:11.120751    2565 patch_prober.go:28] interesting pod/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Jul 30 03:02:11.120980 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:11.120830    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc" podUID="9e928e41-9555-47c4-beaa-ee4c874c0c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Jul 30 03:02:11.814899 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:11.814863    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:02:11.928281 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:11.928245    2565 generic.go:356] "Generic (PLEG): container finished" podID="48569794-7865-465c-8b15-a8e7a21b55fe" containerID="1a420fa0711c315ceaf26c12776abe7aa2f38174e376efb2e5ec3292e281ef3f" exitCode=0
Jul 30 03:02:11.928673 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:11.928338    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fad13d10-9n4bk" event={"ID":"48569794-7865-465c-8b15-a8e7a21b55fe","Type":"ContainerDied","Data":"1a420fa0711c315ceaf26c12776abe7aa2f38174e376efb2e5ec3292e281ef3f"}
Jul 30 03:02:12.119970 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:12.119884    2565 patch_prober.go:28] interesting pod/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Jul 30 03:02:12.119970 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:12.119944    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc" podUID="9e928e41-9555-47c4-beaa-ee4c874c0c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Jul 30 03:02:13.120746 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:13.120717    2565 patch_prober.go:28] interesting pod/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Jul 30 03:02:13.121072 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:13.120769    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc" podUID="9e928e41-9555-47c4-beaa-ee4c874c0c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Jul 30 03:02:13.121747 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:13.121730    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fad13d10-9n4bk"
Jul 30 03:02:13.247318 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:13.247234    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48569794-7865-465c-8b15-a8e7a21b55fe-maas-api-service-ca\") pod \"48569794-7865-465c-8b15-a8e7a21b55fe\" (UID: \"48569794-7865-465c-8b15-a8e7a21b55fe\") "
Jul 30 03:02:13.247559 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:13.247538    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48569794-7865-465c-8b15-a8e7a21b55fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48569794-7865-465c-8b15-a8e7a21b55fe" (UID: "48569794-7865-465c-8b15-a8e7a21b55fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:02:13.348330 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:13.348295    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48569794-7865-465c-8b15-a8e7a21b55fe-maas-api-service-ca\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:02:13.941653 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:13.941614    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fad13d10-9n4bk" event={"ID":"48569794-7865-465c-8b15-a8e7a21b55fe","Type":"ContainerDied","Data":"d40e0fd95f2bb3def0b491e0113d34be97cb06f9870b5a777138a7a9f3526544"}
Jul 30 03:02:13.941653 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:13.941647    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fad13d10-9n4bk"
Jul 30 03:02:13.941869 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:13.941675    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d40e0fd95f2bb3def0b491e0113d34be97cb06f9870b5a777138a7a9f3526544"
Jul 30 03:02:14.124360 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:14.124324    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:02:14.125637 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:14.125609    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:02:19.916335 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:19.916300    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"]
Jul 30 03:02:19.917092 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:19.917050    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg" podUID="fb6200de-b079-4f6b-935c-787acea31c82" containerName="maas-api" containerID="cri-o://6bef8a8368902b348e036b50a19c7f6cc7052b33db8ce8dd1a82ae9661fb1b50" gracePeriod=30
Jul 30 03:02:19.968134 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:19.968099    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp"]
Jul 30 03:02:19.968405 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:19.968370    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp" podUID="cd844156-2889-4ea1-be8d-703a7bcc8001" containerName="payload-processing" containerID="cri-o://46f71fd922adb5d374a23fa14c3cc05a62772c3ed44ff04e4d652f6bddc3d0c9" gracePeriod=30
Jul 30 03:02:19.979956 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:19.978684    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29"]
Jul 30 03:02:19.979956 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:19.978948    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29" podUID="7964f74f-2180-4f18-a8b1-13f2e6f30375" containerName="payload-pre-processing" containerID="cri-o://cc437424a224b87ae7f8399cc923c4fba16b583852a5aeb93b1d7c57c13e5e97" gracePeriod=30
Jul 30 03:02:20.267402 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.267378    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp"
Jul 30 03:02:20.270654 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.270634    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"
Jul 30 03:02:20.286604 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.286579    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29"
Jul 30 03:02:20.317352 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.317317    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lklzz\" (UniqueName: \"kubernetes.io/projected/cd844156-2889-4ea1-be8d-703a7bcc8001-kube-api-access-lklzz\") pod \"cd844156-2889-4ea1-be8d-703a7bcc8001\" (UID: \"cd844156-2889-4ea1-be8d-703a7bcc8001\") "
Jul 30 03:02:20.317544 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.317372    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd844156-2889-4ea1-be8d-703a7bcc8001-plugins-config-volume\") pod \"cd844156-2889-4ea1-be8d-703a7bcc8001\" (UID: \"cd844156-2889-4ea1-be8d-703a7bcc8001\") "
Jul 30 03:02:20.318372 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.318341    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd844156-2889-4ea1-be8d-703a7bcc8001-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cd844156-2889-4ea1-be8d-703a7bcc8001" (UID: "cd844156-2889-4ea1-be8d-703a7bcc8001"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:02:20.320442 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.320411    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd844156-2889-4ea1-be8d-703a7bcc8001-kube-api-access-lklzz" (OuterVolumeSpecName: "kube-api-access-lklzz") pod "cd844156-2889-4ea1-be8d-703a7bcc8001" (UID: "cd844156-2889-4ea1-be8d-703a7bcc8001"). InnerVolumeSpecName "kube-api-access-lklzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:02:20.418979 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.418946    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7964f74f-2180-4f18-a8b1-13f2e6f30375-plugins-config-volume\") pod \"7964f74f-2180-4f18-a8b1-13f2e6f30375\" (UID: \"7964f74f-2180-4f18-a8b1-13f2e6f30375\") "
Jul 30 03:02:20.419198 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.419012    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf6xk\" (UniqueName: \"kubernetes.io/projected/fb6200de-b079-4f6b-935c-787acea31c82-kube-api-access-tf6xk\") pod \"fb6200de-b079-4f6b-935c-787acea31c82\" (UID: \"fb6200de-b079-4f6b-935c-787acea31c82\") "
Jul 30 03:02:20.419198 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.419034    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wwjz\" (UniqueName: \"kubernetes.io/projected/7964f74f-2180-4f18-a8b1-13f2e6f30375-kube-api-access-4wwjz\") pod \"7964f74f-2180-4f18-a8b1-13f2e6f30375\" (UID: \"7964f74f-2180-4f18-a8b1-13f2e6f30375\") "
Jul 30 03:02:20.419198 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.419133    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6200de-b079-4f6b-935c-787acea31c82-maas-api-tls\") pod \"fb6200de-b079-4f6b-935c-787acea31c82\" (UID: \"fb6200de-b079-4f6b-935c-787acea31c82\") "
Jul 30 03:02:20.419198 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.419162    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6200de-b079-4f6b-935c-787acea31c82-openshift-service-ca\") pod \"fb6200de-b079-4f6b-935c-787acea31c82\" (UID: \"fb6200de-b079-4f6b-935c-787acea31c82\") "
Jul 30 03:02:20.419438 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.419388    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7964f74f-2180-4f18-a8b1-13f2e6f30375-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7964f74f-2180-4f18-a8b1-13f2e6f30375" (UID: "7964f74f-2180-4f18-a8b1-13f2e6f30375"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:02:20.419564 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.419541    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lklzz\" (UniqueName: \"kubernetes.io/projected/cd844156-2889-4ea1-be8d-703a7bcc8001-kube-api-access-lklzz\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:02:20.419645 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.419568    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd844156-2889-4ea1-be8d-703a7bcc8001-plugins-config-volume\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:02:20.419645 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.419585    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7964f74f-2180-4f18-a8b1-13f2e6f30375-plugins-config-volume\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:02:20.419645 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.419602    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6200de-b079-4f6b-935c-787acea31c82-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb6200de-b079-4f6b-935c-787acea31c82" (UID: "fb6200de-b079-4f6b-935c-787acea31c82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:02:20.421432 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.421405    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7964f74f-2180-4f18-a8b1-13f2e6f30375-kube-api-access-4wwjz" (OuterVolumeSpecName: "kube-api-access-4wwjz") pod "7964f74f-2180-4f18-a8b1-13f2e6f30375" (UID: "7964f74f-2180-4f18-a8b1-13f2e6f30375"). InnerVolumeSpecName "kube-api-access-4wwjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:02:20.421530 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.421443    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb6200de-b079-4f6b-935c-787acea31c82-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb6200de-b079-4f6b-935c-787acea31c82" (UID: "fb6200de-b079-4f6b-935c-787acea31c82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:02:20.421530 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.421450    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6200de-b079-4f6b-935c-787acea31c82-kube-api-access-tf6xk" (OuterVolumeSpecName: "kube-api-access-tf6xk") pod "fb6200de-b079-4f6b-935c-787acea31c82" (UID: "fb6200de-b079-4f6b-935c-787acea31c82"). InnerVolumeSpecName "kube-api-access-tf6xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:02:20.520487 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.520414    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf6xk\" (UniqueName: \"kubernetes.io/projected/fb6200de-b079-4f6b-935c-787acea31c82-kube-api-access-tf6xk\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:02:20.520487 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.520440    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wwjz\" (UniqueName: \"kubernetes.io/projected/7964f74f-2180-4f18-a8b1-13f2e6f30375-kube-api-access-4wwjz\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:02:20.520487 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.520450    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6200de-b079-4f6b-935c-787acea31c82-maas-api-tls\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:02:20.520487 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.520458    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6200de-b079-4f6b-935c-787acea31c82-openshift-service-ca\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:02:20.995097 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.995058    2565 generic.go:356] "Generic (PLEG): container finished" podID="cd844156-2889-4ea1-be8d-703a7bcc8001" containerID="46f71fd922adb5d374a23fa14c3cc05a62772c3ed44ff04e4d652f6bddc3d0c9" exitCode=0
Jul 30 03:02:20.995571 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.995102    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp"
Jul 30 03:02:20.995571 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.995147    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp" event={"ID":"cd844156-2889-4ea1-be8d-703a7bcc8001","Type":"ContainerDied","Data":"46f71fd922adb5d374a23fa14c3cc05a62772c3ed44ff04e4d652f6bddc3d0c9"}
Jul 30 03:02:20.995571 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.995220    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp" event={"ID":"cd844156-2889-4ea1-be8d-703a7bcc8001","Type":"ContainerDied","Data":"f8de98dca7fd44c49bcafbe99ce344b43cede7f259307b95d169f98f6e845dbe"}
Jul 30 03:02:20.995571 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.995244    2565 scope.go:117] "RemoveContainer" containerID="46f71fd922adb5d374a23fa14c3cc05a62772c3ed44ff04e4d652f6bddc3d0c9"
Jul 30 03:02:20.996639 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.996617    2565 generic.go:356] "Generic (PLEG): container finished" podID="7964f74f-2180-4f18-a8b1-13f2e6f30375" containerID="cc437424a224b87ae7f8399cc923c4fba16b583852a5aeb93b1d7c57c13e5e97" exitCode=0
Jul 30 03:02:20.996728 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.996667    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29"
Jul 30 03:02:20.996792 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.996743    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29" event={"ID":"7964f74f-2180-4f18-a8b1-13f2e6f30375","Type":"ContainerDied","Data":"cc437424a224b87ae7f8399cc923c4fba16b583852a5aeb93b1d7c57c13e5e97"}
Jul 30 03:02:20.996792 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.996786    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29" event={"ID":"7964f74f-2180-4f18-a8b1-13f2e6f30375","Type":"ContainerDied","Data":"219f0efc0aaba3a82a668ff442b259ac4482275ecd2f288166d1e60fb7cfaaef"}
Jul 30 03:02:20.998586 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.998559    2565 generic.go:356] "Generic (PLEG): container finished" podID="fb6200de-b079-4f6b-935c-787acea31c82" containerID="6bef8a8368902b348e036b50a19c7f6cc7052b33db8ce8dd1a82ae9661fb1b50" exitCode=0
Jul 30 03:02:20.998739 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.998602    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg" event={"ID":"fb6200de-b079-4f6b-935c-787acea31c82","Type":"ContainerDied","Data":"6bef8a8368902b348e036b50a19c7f6cc7052b33db8ce8dd1a82ae9661fb1b50"}
Jul 30 03:02:20.998739 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.998615    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"
Jul 30 03:02:20.998739 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:20.998653    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg" event={"ID":"fb6200de-b079-4f6b-935c-787acea31c82","Type":"ContainerDied","Data":"b269be84c9418a3aa6568e4cff18c08b7d5d35518392a67fe1a9e0e7085295ca"}
Jul 30 03:02:21.006070 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:21.006052    2565 scope.go:117] "RemoveContainer" containerID="46f71fd922adb5d374a23fa14c3cc05a62772c3ed44ff04e4d652f6bddc3d0c9"
Jul 30 03:02:21.006374 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:02:21.006353    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f71fd922adb5d374a23fa14c3cc05a62772c3ed44ff04e4d652f6bddc3d0c9\": container with ID starting with 46f71fd922adb5d374a23fa14c3cc05a62772c3ed44ff04e4d652f6bddc3d0c9 not found: ID does not exist" containerID="46f71fd922adb5d374a23fa14c3cc05a62772c3ed44ff04e4d652f6bddc3d0c9"
Jul 30 03:02:21.006439 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:21.006381    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f71fd922adb5d374a23fa14c3cc05a62772c3ed44ff04e4d652f6bddc3d0c9"} err="failed to get container status \"46f71fd922adb5d374a23fa14c3cc05a62772c3ed44ff04e4d652f6bddc3d0c9\": rpc error: code = NotFound desc = could not find container \"46f71fd922adb5d374a23fa14c3cc05a62772c3ed44ff04e4d652f6bddc3d0c9\": container with ID starting with 46f71fd922adb5d374a23fa14c3cc05a62772c3ed44ff04e4d652f6bddc3d0c9 not found: ID does not exist"
Jul 30 03:02:21.006439 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:21.006398    2565 scope.go:117] "RemoveContainer" containerID="cc437424a224b87ae7f8399cc923c4fba16b583852a5aeb93b1d7c57c13e5e97"
Jul 30 03:02:21.015703 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:21.015685    2565 scope.go:117] "RemoveContainer" containerID="cc437424a224b87ae7f8399cc923c4fba16b583852a5aeb93b1d7c57c13e5e97"
Jul 30 03:02:21.016050 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:02:21.016024    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc437424a224b87ae7f8399cc923c4fba16b583852a5aeb93b1d7c57c13e5e97\": container with ID starting with cc437424a224b87ae7f8399cc923c4fba16b583852a5aeb93b1d7c57c13e5e97 not found: ID does not exist" containerID="cc437424a224b87ae7f8399cc923c4fba16b583852a5aeb93b1d7c57c13e5e97"
Jul 30 03:02:21.016129 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:21.016058    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc437424a224b87ae7f8399cc923c4fba16b583852a5aeb93b1d7c57c13e5e97"} err="failed to get container status \"cc437424a224b87ae7f8399cc923c4fba16b583852a5aeb93b1d7c57c13e5e97\": rpc error: code = NotFound desc = could not find container \"cc437424a224b87ae7f8399cc923c4fba16b583852a5aeb93b1d7c57c13e5e97\": container with ID starting with cc437424a224b87ae7f8399cc923c4fba16b583852a5aeb93b1d7c57c13e5e97 not found: ID does not exist"
Jul 30 03:02:21.016129 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:21.016082    2565 scope.go:117] "RemoveContainer" containerID="6bef8a8368902b348e036b50a19c7f6cc7052b33db8ce8dd1a82ae9661fb1b50"
Jul 30 03:02:21.019571 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:21.019032    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"]
Jul 30 03:02:21.022668 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:21.022647    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fad13d10-66b9dbc9c9-vzjrg"]
Jul 30 03:02:21.025299 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:21.025281    2565 scope.go:117] "RemoveContainer" containerID="6bef8a8368902b348e036b50a19c7f6cc7052b33db8ce8dd1a82ae9661fb1b50"
Jul 30 03:02:21.025546 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:02:21.025526    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bef8a8368902b348e036b50a19c7f6cc7052b33db8ce8dd1a82ae9661fb1b50\": container with ID starting with 6bef8a8368902b348e036b50a19c7f6cc7052b33db8ce8dd1a82ae9661fb1b50 not found: ID does not exist" containerID="6bef8a8368902b348e036b50a19c7f6cc7052b33db8ce8dd1a82ae9661fb1b50"
Jul 30 03:02:21.025589 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:21.025553    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bef8a8368902b348e036b50a19c7f6cc7052b33db8ce8dd1a82ae9661fb1b50"} err="failed to get container status \"6bef8a8368902b348e036b50a19c7f6cc7052b33db8ce8dd1a82ae9661fb1b50\": rpc error: code = NotFound desc = could not find container \"6bef8a8368902b348e036b50a19c7f6cc7052b33db8ce8dd1a82ae9661fb1b50\": container with ID starting with 6bef8a8368902b348e036b50a19c7f6cc7052b33db8ce8dd1a82ae9661fb1b50 not found: ID does not exist"
Jul 30 03:02:21.032307 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:21.032284    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29"]
Jul 30 03:02:21.035725 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:21.035706    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fad13d10-74c5dcb-d8v29"]
Jul 30 03:02:21.048909 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:21.048785    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp"]
Jul 30 03:02:21.053869 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:21.053839    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fad13d10-7cb98d457d-fgrdp"]
Jul 30 03:02:22.801771 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:22.801737    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7964f74f-2180-4f18-a8b1-13f2e6f30375" path="/var/lib/kubelet/pods/7964f74f-2180-4f18-a8b1-13f2e6f30375/volumes"
Jul 30 03:02:22.802252 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:22.802093    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd844156-2889-4ea1-be8d-703a7bcc8001" path="/var/lib/kubelet/pods/cd844156-2889-4ea1-be8d-703a7bcc8001/volumes"
Jul 30 03:02:22.802505 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:22.802486    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb6200de-b079-4f6b-935c-787acea31c82" path="/var/lib/kubelet/pods/fb6200de-b079-4f6b-935c-787acea31c82/volumes"
Jul 30 03:02:24.596044 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:24.596008    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fad13d10-9n4bk"]
Jul 30 03:02:24.598150 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:24.598129    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fad13d10-9n4bk"]
Jul 30 03:02:24.800873 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:24.800839    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48569794-7865-465c-8b15-a8e7a21b55fe" path="/var/lib/kubelet/pods/48569794-7865-465c-8b15-a8e7a21b55fe/volumes"
Jul 30 03:02:31.165183 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:31.165139    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"]
Jul 30 03:02:31.165614 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:31.165365    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc" podUID="9e928e41-9555-47c4-beaa-ee4c874c0c0c" containerName="istio-proxy" containerID="cri-o://dfc2295f10c3eca375a1d61004b2524ef79ac25076244f3bb6d56dc304973228" gracePeriod=30
Jul 30 03:02:36.417580 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.417550    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:02:36.574142 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.574112    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-envoy\") pod \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") "
Jul 30 03:02:36.574142 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.574147    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-data\") pod \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") "
Jul 30 03:02:36.574442 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.574198    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-credential-socket\") pod \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") "
Jul 30 03:02:36.574442 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.574240    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nwnw\" (UniqueName: \"kubernetes.io/projected/9e928e41-9555-47c4-beaa-ee4c874c0c0c-kube-api-access-7nwnw\") pod \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") "
Jul 30 03:02:36.574442 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.574271    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-podinfo\") pod \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") "
Jul 30 03:02:36.574442 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.574308    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-token\") pod \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") "
Jul 30 03:02:36.574655 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.574451    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-workload-socket\") pod \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") "
Jul 30 03:02:36.574655 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.574486    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istiod-ca-cert\") pod \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") "
Jul 30 03:02:36.574655 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.574545    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-workload-certs\") pod \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\" (UID: \"9e928e41-9555-47c4-beaa-ee4c874c0c0c\") "
Jul 30 03:02:36.574655 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.574594    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9e928e41-9555-47c4-beaa-ee4c874c0c0c" (UID: "9e928e41-9555-47c4-beaa-ee4c874c0c0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:02:36.574862 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.574793    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9e928e41-9555-47c4-beaa-ee4c874c0c0c" (UID: "9e928e41-9555-47c4-beaa-ee4c874c0c0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:02:36.574924 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.574902    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-credential-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:02:36.574924 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.574919    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-workload-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:02:36.575209 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.575182    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9e928e41-9555-47c4-beaa-ee4c874c0c0c" (UID: "9e928e41-9555-47c4-beaa-ee4c874c0c0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:02:36.575308 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.575225    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9e928e41-9555-47c4-beaa-ee4c874c0c0c" (UID: "9e928e41-9555-47c4-beaa-ee4c874c0c0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:02:36.575308 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.575257    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-data" (OuterVolumeSpecName: "istio-data") pod "9e928e41-9555-47c4-beaa-ee4c874c0c0c" (UID: "9e928e41-9555-47c4-beaa-ee4c874c0c0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:02:36.576544 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.576520    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9e928e41-9555-47c4-beaa-ee4c874c0c0c" (UID: "9e928e41-9555-47c4-beaa-ee4c874c0c0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:02:36.576703 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.576687    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-token" (OuterVolumeSpecName: "istio-token") pod "9e928e41-9555-47c4-beaa-ee4c874c0c0c" (UID: "9e928e41-9555-47c4-beaa-ee4c874c0c0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:02:36.576982 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.576956    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e928e41-9555-47c4-beaa-ee4c874c0c0c-kube-api-access-7nwnw" (OuterVolumeSpecName: "kube-api-access-7nwnw") pod "9e928e41-9555-47c4-beaa-ee4c874c0c0c" (UID: "9e928e41-9555-47c4-beaa-ee4c874c0c0c"). InnerVolumeSpecName "kube-api-access-7nwnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:02:36.577033 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.576969    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9e928e41-9555-47c4-beaa-ee4c874c0c0c" (UID: "9e928e41-9555-47c4-beaa-ee4c874c0c0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 30 03:02:36.676343 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.676309    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-workload-certs\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:02:36.676343 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.676336    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-envoy\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:02:36.676343 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.676344    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-data\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:02:36.676578 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.676353    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nwnw\" (UniqueName: \"kubernetes.io/projected/9e928e41-9555-47c4-beaa-ee4c874c0c0c-kube-api-access-7nwnw\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:02:36.676578 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.676362    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-podinfo\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:02:36.676578 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.676371    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istio-token\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:02:36.676578 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:36.676378    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e928e41-9555-47c4-beaa-ee4c874c0c0c-istiod-ca-cert\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:02:37.100968 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:37.100929    2565 generic.go:356] "Generic (PLEG): container finished" podID="9e928e41-9555-47c4-beaa-ee4c874c0c0c" containerID="dfc2295f10c3eca375a1d61004b2524ef79ac25076244f3bb6d56dc304973228" exitCode=0
Jul 30 03:02:37.101181 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:37.100982    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"
Jul 30 03:02:37.101181 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:37.101022    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc" event={"ID":"9e928e41-9555-47c4-beaa-ee4c874c0c0c","Type":"ContainerDied","Data":"dfc2295f10c3eca375a1d61004b2524ef79ac25076244f3bb6d56dc304973228"}
Jul 30 03:02:37.101181 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:37.101083    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc" event={"ID":"9e928e41-9555-47c4-beaa-ee4c874c0c0c","Type":"ContainerDied","Data":"0cc6324c86379b4c094145fb655cc078194aed7f030c598b81a56b3416c1d6a8"}
Jul 30 03:02:37.101181 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:37.101108    2565 scope.go:117] "RemoveContainer" containerID="dfc2295f10c3eca375a1d61004b2524ef79ac25076244f3bb6d56dc304973228"
Jul 30 03:02:37.110502 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:37.110484    2565 scope.go:117] "RemoveContainer" containerID="d1137f59e86be0f4c6678f2180e7366f03b964d904ac484f70eb79117d163c8d"
Jul 30 03:02:37.119745 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:37.119724    2565 scope.go:117] "RemoveContainer" containerID="dfc2295f10c3eca375a1d61004b2524ef79ac25076244f3bb6d56dc304973228"
Jul 30 03:02:37.119836 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:37.119820    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"]
Jul 30 03:02:37.120043 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:02:37.120024    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfc2295f10c3eca375a1d61004b2524ef79ac25076244f3bb6d56dc304973228\": container with ID starting with dfc2295f10c3eca375a1d61004b2524ef79ac25076244f3bb6d56dc304973228 not found: ID does not exist" containerID="dfc2295f10c3eca375a1d61004b2524ef79ac25076244f3bb6d56dc304973228"
Jul 30 03:02:37.120112 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:37.120051    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfc2295f10c3eca375a1d61004b2524ef79ac25076244f3bb6d56dc304973228"} err="failed to get container status \"dfc2295f10c3eca375a1d61004b2524ef79ac25076244f3bb6d56dc304973228\": rpc error: code = NotFound desc = could not find container \"dfc2295f10c3eca375a1d61004b2524ef79ac25076244f3bb6d56dc304973228\": container with ID starting with dfc2295f10c3eca375a1d61004b2524ef79ac25076244f3bb6d56dc304973228 not found: ID does not exist"
Jul 30 03:02:37.120112 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:37.120067    2565 scope.go:117] "RemoveContainer" containerID="d1137f59e86be0f4c6678f2180e7366f03b964d904ac484f70eb79117d163c8d"
Jul 30 03:02:37.120332 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:02:37.120314    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1137f59e86be0f4c6678f2180e7366f03b964d904ac484f70eb79117d163c8d\": container with ID starting with d1137f59e86be0f4c6678f2180e7366f03b964d904ac484f70eb79117d163c8d not found: ID does not exist" containerID="d1137f59e86be0f4c6678f2180e7366f03b964d904ac484f70eb79117d163c8d"
Jul 30 03:02:37.120392 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:37.120341    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1137f59e86be0f4c6678f2180e7366f03b964d904ac484f70eb79117d163c8d"} err="failed to get container status \"d1137f59e86be0f4c6678f2180e7366f03b964d904ac484f70eb79117d163c8d\": rpc error: code = NotFound desc = could not find container \"d1137f59e86be0f4c6678f2180e7366f03b964d904ac484f70eb79117d163c8d\": container with ID starting with d1137f59e86be0f4c6678f2180e7366f03b964d904ac484f70eb79117d163c8d not found: ID does not exist"
Jul 30 03:02:37.122107 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:37.122088    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fad13d10-openshift-default-85bfd8f476-9vxvc"]
Jul 30 03:02:38.801097 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:02:38.801051    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e928e41-9555-47c4-beaa-ee4c874c0c0c" path="/var/lib/kubelet/pods/9e928e41-9555-47c4-beaa-ee4c874c0c0c/volumes"
Jul 30 03:03:01.072621 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:03:01.072576    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz"]
Jul 30 03:03:01.073127 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:03:01.072793    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz" podUID="4e0030d3-06ab-49b6-b3d0-418364956512" containerName="manager" containerID="cri-o://cd42da0bb2d02d57547d2d79256e9f2ee75d457b1004c07fef7767d1c459f634" gracePeriod=10
Jul 30 03:03:02.229824 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:03:02.229799    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz"
Jul 30 03:03:02.253840 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:03:02.253810    2565 generic.go:356] "Generic (PLEG): container finished" podID="4e0030d3-06ab-49b6-b3d0-418364956512" containerID="cd42da0bb2d02d57547d2d79256e9f2ee75d457b1004c07fef7767d1c459f634" exitCode=0
Jul 30 03:03:02.253994 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:03:02.253865    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz"
Jul 30 03:03:02.253994 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:03:02.253907    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz" event={"ID":"4e0030d3-06ab-49b6-b3d0-418364956512","Type":"ContainerDied","Data":"cd42da0bb2d02d57547d2d79256e9f2ee75d457b1004c07fef7767d1c459f634"}
Jul 30 03:03:02.254105 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:03:02.253998    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz" event={"ID":"4e0030d3-06ab-49b6-b3d0-418364956512","Type":"ContainerDied","Data":"82226b88b75ea5a35a0e57b9e6089dd838fca3ee079febbe2ee6ea8755e6abd5"}
Jul 30 03:03:02.254105 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:03:02.254023    2565 scope.go:117] "RemoveContainer" containerID="cd42da0bb2d02d57547d2d79256e9f2ee75d457b1004c07fef7767d1c459f634"
Jul 30 03:03:02.268530 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:03:02.267700    2565 scope.go:117] "RemoveContainer" containerID="cd42da0bb2d02d57547d2d79256e9f2ee75d457b1004c07fef7767d1c459f634"
Jul 30 03:03:02.269541 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:03:02.269515    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd42da0bb2d02d57547d2d79256e9f2ee75d457b1004c07fef7767d1c459f634\": container with ID starting with cd42da0bb2d02d57547d2d79256e9f2ee75d457b1004c07fef7767d1c459f634 not found: ID does not exist" containerID="cd42da0bb2d02d57547d2d79256e9f2ee75d457b1004c07fef7767d1c459f634"
Jul 30 03:03:02.269671 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:03:02.269550    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd42da0bb2d02d57547d2d79256e9f2ee75d457b1004c07fef7767d1c459f634"} err="failed to get container status \"cd42da0bb2d02d57547d2d79256e9f2ee75d457b1004c07fef7767d1c459f634\": rpc error: code = NotFound desc = could not find container \"cd42da0bb2d02d57547d2d79256e9f2ee75d457b1004c07fef7767d1c459f634\": container with ID starting with cd42da0bb2d02d57547d2d79256e9f2ee75d457b1004c07fef7767d1c459f634 not found: ID does not exist"
Jul 30 03:03:02.295071 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:03:02.295050    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e0030d3-06ab-49b6-b3d0-418364956512-extensions-socket-volume\") pod \"4e0030d3-06ab-49b6-b3d0-418364956512\" (UID: \"4e0030d3-06ab-49b6-b3d0-418364956512\") "
Jul 30 03:03:02.295262 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:03:02.295148    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6prgj\" (UniqueName: \"kubernetes.io/projected/4e0030d3-06ab-49b6-b3d0-418364956512-kube-api-access-6prgj\") pod \"4e0030d3-06ab-49b6-b3d0-418364956512\" (UID: \"4e0030d3-06ab-49b6-b3d0-418364956512\") "
Jul 30 03:03:02.295495 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:03:02.295471    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0030d3-06ab-49b6-b3d0-418364956512-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4e0030d3-06ab-49b6-b3d0-418364956512" (UID: "4e0030d3-06ab-49b6-b3d0-418364956512"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:03:02.297282 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:03:02.297257    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0030d3-06ab-49b6-b3d0-418364956512-kube-api-access-6prgj" (OuterVolumeSpecName: "kube-api-access-6prgj") pod "4e0030d3-06ab-49b6-b3d0-418364956512" (UID: "4e0030d3-06ab-49b6-b3d0-418364956512"). InnerVolumeSpecName "kube-api-access-6prgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:03:02.396471 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:03:02.396397    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6prgj\" (UniqueName: \"kubernetes.io/projected/4e0030d3-06ab-49b6-b3d0-418364956512-kube-api-access-6prgj\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:03:02.396471 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:03:02.396423    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e0030d3-06ab-49b6-b3d0-418364956512-extensions-socket-volume\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:03:02.577521 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:03:02.577490    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz"]
Jul 30 03:03:02.581848 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:03:02.581821    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47gkz"]
Jul 30 03:03:02.800312 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:03:02.800281    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e0030d3-06ab-49b6-b3d0-418364956512" path="/var/lib/kubelet/pods/4e0030d3-06ab-49b6-b3d0-418364956512/volumes"
Jul 30 03:04:16.700767 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.700733    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djc9v"]
Jul 30 03:04:16.701291 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701091    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd844156-2889-4ea1-be8d-703a7bcc8001" containerName="payload-processing"
Jul 30 03:04:16.701291 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701104    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd844156-2889-4ea1-be8d-703a7bcc8001" containerName="payload-processing"
Jul 30 03:04:16.701291 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701114    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48569794-7865-465c-8b15-a8e7a21b55fe" containerName="revoke-keys"
Jul 30 03:04:16.701291 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701120    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="48569794-7865-465c-8b15-a8e7a21b55fe" containerName="revoke-keys"
Jul 30 03:04:16.701291 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701140    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e0030d3-06ab-49b6-b3d0-418364956512" containerName="manager"
Jul 30 03:04:16.701291 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701145    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e0030d3-06ab-49b6-b3d0-418364956512" containerName="manager"
Jul 30 03:04:16.701291 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701153    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb6200de-b079-4f6b-935c-787acea31c82" containerName="maas-api"
Jul 30 03:04:16.701291 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701158    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb6200de-b079-4f6b-935c-787acea31c82" containerName="maas-api"
Jul 30 03:04:16.701291 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701179    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7964f74f-2180-4f18-a8b1-13f2e6f30375" containerName="payload-pre-processing"
Jul 30 03:04:16.701291 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701184    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7964f74f-2180-4f18-a8b1-13f2e6f30375" containerName="payload-pre-processing"
Jul 30 03:04:16.701291 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701190    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e928e41-9555-47c4-beaa-ee4c874c0c0c" containerName="istio-proxy"
Jul 30 03:04:16.701291 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701195    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e928e41-9555-47c4-beaa-ee4c874c0c0c" containerName="istio-proxy"
Jul 30 03:04:16.701291 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701201    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e928e41-9555-47c4-beaa-ee4c874c0c0c" containerName="istio-proxy"
Jul 30 03:04:16.701291 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701206    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e928e41-9555-47c4-beaa-ee4c874c0c0c" containerName="istio-proxy"
Jul 30 03:04:16.701291 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701278    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e0030d3-06ab-49b6-b3d0-418364956512" containerName="manager"
Jul 30 03:04:16.701291 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701285    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e928e41-9555-47c4-beaa-ee4c874c0c0c" containerName="istio-proxy"
Jul 30 03:04:16.701291 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701290    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb6200de-b079-4f6b-935c-787acea31c82" containerName="maas-api"
Jul 30 03:04:16.701291 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701296    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e928e41-9555-47c4-beaa-ee4c874c0c0c" containerName="istio-proxy"
Jul 30 03:04:16.701291 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701302    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7964f74f-2180-4f18-a8b1-13f2e6f30375" containerName="payload-pre-processing"
Jul 30 03:04:16.701920 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701307    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd844156-2889-4ea1-be8d-703a7bcc8001" containerName="payload-processing"
Jul 30 03:04:16.701920 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.701315    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="48569794-7865-465c-8b15-a8e7a21b55fe" containerName="revoke-keys"
Jul 30 03:04:16.704293 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.704277    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djc9v"
Jul 30 03:04:16.707819 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.707794    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kgss5\""
Jul 30 03:04:16.716440 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.716415    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djc9v"]
Jul 30 03:04:16.866811 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.866777    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33d3c02b-b591-4da0-9178-3cfb7eafa488-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-djc9v\" (UID: \"33d3c02b-b591-4da0-9178-3cfb7eafa488\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djc9v"
Jul 30 03:04:16.866983 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.866845    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w57kf\" (UniqueName: \"kubernetes.io/projected/33d3c02b-b591-4da0-9178-3cfb7eafa488-kube-api-access-w57kf\") pod \"kuadrant-operator-controller-manager-86878c769f-djc9v\" (UID: \"33d3c02b-b591-4da0-9178-3cfb7eafa488\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djc9v"
Jul 30 03:04:16.967946 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.967863    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33d3c02b-b591-4da0-9178-3cfb7eafa488-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-djc9v\" (UID: \"33d3c02b-b591-4da0-9178-3cfb7eafa488\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djc9v"
Jul 30 03:04:16.967946 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.967927    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w57kf\" (UniqueName: \"kubernetes.io/projected/33d3c02b-b591-4da0-9178-3cfb7eafa488-kube-api-access-w57kf\") pod \"kuadrant-operator-controller-manager-86878c769f-djc9v\" (UID: \"33d3c02b-b591-4da0-9178-3cfb7eafa488\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djc9v"
Jul 30 03:04:16.968252 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.968231    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33d3c02b-b591-4da0-9178-3cfb7eafa488-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-djc9v\" (UID: \"33d3c02b-b591-4da0-9178-3cfb7eafa488\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djc9v"
Jul 30 03:04:16.978401 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:16.978380    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w57kf\" (UniqueName: \"kubernetes.io/projected/33d3c02b-b591-4da0-9178-3cfb7eafa488-kube-api-access-w57kf\") pod \"kuadrant-operator-controller-manager-86878c769f-djc9v\" (UID: \"33d3c02b-b591-4da0-9178-3cfb7eafa488\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djc9v"
Jul 30 03:04:17.014349 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:17.014324    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djc9v"
Jul 30 03:04:17.157903 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:17.157878    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djc9v"]
Jul 30 03:04:17.159656 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:04:17.159622    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d3c02b_b591_4da0_9178_3cfb7eafa488.slice/crio-bc3238ba82646a53779733c126b2e60c0dbb5c19636a9ce01adfd6bfea539501 WatchSource:0}: Error finding container bc3238ba82646a53779733c126b2e60c0dbb5c19636a9ce01adfd6bfea539501: Status 404 returned error can't find the container with id bc3238ba82646a53779733c126b2e60c0dbb5c19636a9ce01adfd6bfea539501
Jul 30 03:04:17.689884 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:17.689846    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djc9v" event={"ID":"33d3c02b-b591-4da0-9178-3cfb7eafa488","Type":"ContainerStarted","Data":"080bf5d1c0daadb63d040a51af56c23b4f6e8d7703e4e18eddb51b665f630075"}
Jul 30 03:04:17.690065 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:17.689908    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djc9v" event={"ID":"33d3c02b-b591-4da0-9178-3cfb7eafa488","Type":"ContainerStarted","Data":"bc3238ba82646a53779733c126b2e60c0dbb5c19636a9ce01adfd6bfea539501"}
Jul 30 03:04:17.690065 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:17.689991    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djc9v"
Jul 30 03:04:17.717264 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:17.717209    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djc9v" podStartSLOduration=1.717193497 podStartE2EDuration="1.717193497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:04:17.715466156 +0000 UTC m=+2293.362249558" watchObservedRunningTime="2026-07-30 03:04:17.717193497 +0000 UTC m=+2293.363976899"
Jul 30 03:04:28.698577 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:28.698543    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djc9v"
Jul 30 03:04:35.644741 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.644702    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"]
Jul 30 03:04:35.648563 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.648539    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.652102 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.652078    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-adb22f94-openshift-default-dockercfg-2h9wb\""
Jul 30 03:04:35.692611 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.692580    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"]
Jul 30 03:04:35.737043 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.737009    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-token\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.737043 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.737046    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-workload-socket\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.737286 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.737063    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-credential-socket\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.737286 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.737115    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gs4d\" (UniqueName: \"kubernetes.io/projected/594025e4-2c47-4fec-a6dc-c4ce28958b01-kube-api-access-5gs4d\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.737286 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.737150    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-podinfo\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.737286 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.737229    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-envoy\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.737286 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.737247    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-workload-certs\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.737286 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.737271    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-data\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.737286 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.737287    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/594025e4-2c47-4fec-a6dc-c4ce28958b01-istiod-ca-cert\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.838189 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.838130    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-credential-socket\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.838359 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.838202    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gs4d\" (UniqueName: \"kubernetes.io/projected/594025e4-2c47-4fec-a6dc-c4ce28958b01-kube-api-access-5gs4d\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.838359 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.838237    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-podinfo\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.838359 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.838319    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-envoy\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.838359 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.838345    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-workload-certs\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.838577 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.838383    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-data\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.838577 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.838428    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/594025e4-2c47-4fec-a6dc-c4ce28958b01-istiod-ca-cert\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.838577 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.838490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-token\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.838577 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.838517    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-workload-socket\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.838577 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.838514    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-credential-socket\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.838839 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.838786    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-workload-certs\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.839087 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.839032    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-data\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.839418 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.839398    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-workload-socket\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.839550 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.839529    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/594025e4-2c47-4fec-a6dc-c4ce28958b01-istiod-ca-cert\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.840707 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.840685    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-envoy\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.841079 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.841061    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-podinfo\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.854234 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.854211    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-token\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.854346 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.854296    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gs4d\" (UniqueName: \"kubernetes.io/projected/594025e4-2c47-4fec-a6dc-c4ce28958b01-kube-api-access-5gs4d\") pod \"e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") " pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:35.958457 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:35.958369    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:36.117303 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:36.114817    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"]
Jul 30 03:04:36.119083 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:04:36.119057    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594025e4_2c47_4fec_a6dc_c4ce28958b01.slice/crio-c9ff6bd4e81f562f888216981315d7e432ea6ad1758c6aaee46c25f363537afe WatchSource:0}: Error finding container c9ff6bd4e81f562f888216981315d7e432ea6ad1758c6aaee46c25f363537afe: Status 404 returned error can't find the container with id c9ff6bd4e81f562f888216981315d7e432ea6ad1758c6aaee46c25f363537afe
Jul 30 03:04:36.121183 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:36.121125    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:04:36.121285 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:36.121244    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:04:36.121350 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:36.121290    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:04:36.801119 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:36.801068    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck" event={"ID":"594025e4-2c47-4fec-a6dc-c4ce28958b01","Type":"ContainerStarted","Data":"1a73f3dca6b98ba7836cae6b2ae12c0f2f5622ed9f47cde3ab12027d71f27640"}
Jul 30 03:04:36.801495 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:36.801130    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck" event={"ID":"594025e4-2c47-4fec-a6dc-c4ce28958b01","Type":"ContainerStarted","Data":"c9ff6bd4e81f562f888216981315d7e432ea6ad1758c6aaee46c25f363537afe"}
Jul 30 03:04:36.834734 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:36.834683    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck" podStartSLOduration=1.8346679799999999 podStartE2EDuration="1.83466798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:04:36.834298878 +0000 UTC m=+2312.481082284" watchObservedRunningTime="2026-07-30 03:04:36.83466798 +0000 UTC m=+2312.481451385"
Jul 30 03:04:36.958555 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:36.958516    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:37.964043 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:37.964010    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:38.814673 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:38.814640    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:38.815542 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:38.815522    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:39.072687 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:39.072602    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"]
Jul 30 03:04:39.076004 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:39.075983    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"
Jul 30 03:04:39.079440 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:39.079418    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-adb22f94-serving-cert\""
Jul 30 03:04:39.108439 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:39.108414    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"]
Jul 30 03:04:39.167014 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:39.166977    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-openshift-service-ca\") pod \"maas-api-e2e-mt-adb22f94-979695fd7-q98pw\" (UID: \"9a77ddb9-9d12-47fb-a432-661b1c8cbaa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"
Jul 30 03:04:39.167232 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:39.167036    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c6jt\" (UniqueName: \"kubernetes.io/projected/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-kube-api-access-5c6jt\") pod \"maas-api-e2e-mt-adb22f94-979695fd7-q98pw\" (UID: \"9a77ddb9-9d12-47fb-a432-661b1c8cbaa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"
Jul 30 03:04:39.167232 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:39.167076    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-maas-api-tls\") pod \"maas-api-e2e-mt-adb22f94-979695fd7-q98pw\" (UID: \"9a77ddb9-9d12-47fb-a432-661b1c8cbaa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"
Jul 30 03:04:39.268321 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:39.268282    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-openshift-service-ca\") pod \"maas-api-e2e-mt-adb22f94-979695fd7-q98pw\" (UID: \"9a77ddb9-9d12-47fb-a432-661b1c8cbaa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"
Jul 30 03:04:39.268502 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:39.268344    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c6jt\" (UniqueName: \"kubernetes.io/projected/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-kube-api-access-5c6jt\") pod \"maas-api-e2e-mt-adb22f94-979695fd7-q98pw\" (UID: \"9a77ddb9-9d12-47fb-a432-661b1c8cbaa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"
Jul 30 03:04:39.268502 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:39.268378    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-maas-api-tls\") pod \"maas-api-e2e-mt-adb22f94-979695fd7-q98pw\" (UID: \"9a77ddb9-9d12-47fb-a432-661b1c8cbaa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"
Jul 30 03:04:39.268502 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:04:39.268496    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-serving-cert: secret "maas-api-e2e-mt-adb22f94-serving-cert" not found
Jul 30 03:04:39.268614 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:04:39.268564    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-maas-api-tls podName:9a77ddb9-9d12-47fb-a432-661b1c8cbaa9 nodeName:}" failed. No retries permitted until 2026-07-30 03:04:39.768547328 +0000 UTC m=+2315.415330711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-maas-api-tls") pod "maas-api-e2e-mt-adb22f94-979695fd7-q98pw" (UID: "9a77ddb9-9d12-47fb-a432-661b1c8cbaa9") : secret "maas-api-e2e-mt-adb22f94-serving-cert" not found
Jul 30 03:04:39.268963 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:39.268944    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-openshift-service-ca\") pod \"maas-api-e2e-mt-adb22f94-979695fd7-q98pw\" (UID: \"9a77ddb9-9d12-47fb-a432-661b1c8cbaa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"
Jul 30 03:04:39.301977 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:39.301942    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c6jt\" (UniqueName: \"kubernetes.io/projected/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-kube-api-access-5c6jt\") pod \"maas-api-e2e-mt-adb22f94-979695fd7-q98pw\" (UID: \"9a77ddb9-9d12-47fb-a432-661b1c8cbaa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"
Jul 30 03:04:39.772829 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:39.772795    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-maas-api-tls\") pod \"maas-api-e2e-mt-adb22f94-979695fd7-q98pw\" (UID: \"9a77ddb9-9d12-47fb-a432-661b1c8cbaa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"
Jul 30 03:04:39.775861 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:39.775827    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-maas-api-tls\") pod \"maas-api-e2e-mt-adb22f94-979695fd7-q98pw\" (UID: \"9a77ddb9-9d12-47fb-a432-661b1c8cbaa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"
Jul 30 03:04:39.987642 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:39.987613    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"
Jul 30 03:04:40.147663 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:40.147637    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"]
Jul 30 03:04:40.150020 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:04:40.149993    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a77ddb9_9d12_47fb_a432_661b1c8cbaa9.slice/crio-23730778247e0ace912a2ca44cf56f3a1b279384b4c1ca9e29d2d7294045d3e7 WatchSource:0}: Error finding container 23730778247e0ace912a2ca44cf56f3a1b279384b4c1ca9e29d2d7294045d3e7: Status 404 returned error can't find the container with id 23730778247e0ace912a2ca44cf56f3a1b279384b4c1ca9e29d2d7294045d3e7
Jul 30 03:04:40.151976 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:40.151954    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 03:04:40.483440 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:40.483349    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8"]
Jul 30 03:04:40.488033 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:40.488012    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8"
Jul 30 03:04:40.494530 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:04:40.494499    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8\" is forbidden: User \"system:node:ip-10-0-132-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-132-89.ec2.internal' and this object" podUID="ed7a9c3a-35ec-40ba-8540-14136d018ccf" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8"
Jul 30 03:04:40.494669 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:04:40.494521    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"payload-processing-e2e-mt-adb22f94-dockercfg-g5bhw\" is forbidden: User \"system:node:ip-10-0-132-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-132-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-adb22f94-dockercfg-g5bhw\"" type="*v1.Secret"
Jul 30 03:04:40.494767 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:04:40.494748    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"payload-processing-plugins-e2e-mt-adb22f94\" is forbidden: User \"system:node:ip-10-0-132-89.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-132-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-adb22f94\"" type="*v1.ConfigMap"
Jul 30 03:04:40.566646 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:40.566612    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8"]
Jul 30 03:04:40.581396 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:40.581367    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7a9c3a-35ec-40ba-8540-14136d018ccf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8\" (UID: \"ed7a9c3a-35ec-40ba-8540-14136d018ccf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8"
Jul 30 03:04:40.581558 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:40.581472    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92bg6\" (UniqueName: \"kubernetes.io/projected/ed7a9c3a-35ec-40ba-8540-14136d018ccf-kube-api-access-92bg6\") pod \"payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8\" (UID: \"ed7a9c3a-35ec-40ba-8540-14136d018ccf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8"
Jul 30 03:04:40.681943 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:40.681912    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92bg6\" (UniqueName: \"kubernetes.io/projected/ed7a9c3a-35ec-40ba-8540-14136d018ccf-kube-api-access-92bg6\") pod \"payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8\" (UID: \"ed7a9c3a-35ec-40ba-8540-14136d018ccf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8"
Jul 30 03:04:40.682106 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:40.681984    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7a9c3a-35ec-40ba-8540-14136d018ccf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8\" (UID: \"ed7a9c3a-35ec-40ba-8540-14136d018ccf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8"
Jul 30 03:04:40.702728 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:40.702689    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92bg6\" (UniqueName: \"kubernetes.io/projected/ed7a9c3a-35ec-40ba-8540-14136d018ccf-kube-api-access-92bg6\") pod \"payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8\" (UID: \"ed7a9c3a-35ec-40ba-8540-14136d018ccf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8"
Jul 30 03:04:40.721420 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:40.721390    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9"]
Jul 30 03:04:40.724930 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:40.724914    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9"
Jul 30 03:04:40.767059 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:40.766973    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9"]
Jul 30 03:04:40.782598 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:40.782559    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4152435-679b-4cba-9f41-f8692f90c305-plugins-config-volume\") pod \"payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9\" (UID: \"b4152435-679b-4cba-9f41-f8692f90c305\") " pod="openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9"
Jul 30 03:04:40.782778 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:40.782628    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lcsl\" (UniqueName: \"kubernetes.io/projected/b4152435-679b-4cba-9f41-f8692f90c305-kube-api-access-7lcsl\") pod \"payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9\" (UID: \"b4152435-679b-4cba-9f41-f8692f90c305\") " pod="openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9"
Jul 30 03:04:40.829809 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:40.829755    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw" event={"ID":"9a77ddb9-9d12-47fb-a432-661b1c8cbaa9","Type":"ContainerStarted","Data":"23730778247e0ace912a2ca44cf56f3a1b279384b4c1ca9e29d2d7294045d3e7"}
Jul 30 03:04:40.883566 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:40.883524    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4152435-679b-4cba-9f41-f8692f90c305-plugins-config-volume\") pod \"payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9\" (UID: \"b4152435-679b-4cba-9f41-f8692f90c305\") " pod="openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9"
Jul 30 03:04:40.883742 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:40.883582    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lcsl\" (UniqueName: \"kubernetes.io/projected/b4152435-679b-4cba-9f41-f8692f90c305-kube-api-access-7lcsl\") pod \"payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9\" (UID: \"b4152435-679b-4cba-9f41-f8692f90c305\") " pod="openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9"
Jul 30 03:04:40.910482 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:40.910449    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lcsl\" (UniqueName: \"kubernetes.io/projected/b4152435-679b-4cba-9f41-f8692f90c305-kube-api-access-7lcsl\") pod \"payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9\" (UID: \"b4152435-679b-4cba-9f41-f8692f90c305\") " pod="openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9"
Jul 30 03:04:41.508508 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:41.508479    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-adb22f94-dockercfg-g5bhw\""
Jul 30 03:04:41.682772 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:04:41.682738    2565 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-mt-adb22f94: failed to sync configmap cache: timed out waiting for the condition
Jul 30 03:04:41.682946 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:04:41.682818    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed7a9c3a-35ec-40ba-8540-14136d018ccf-plugins-config-volume podName:ed7a9c3a-35ec-40ba-8540-14136d018ccf nodeName:}" failed. No retries permitted until 2026-07-30 03:04:42.182800872 +0000 UTC m=+2317.829584259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/ed7a9c3a-35ec-40ba-8540-14136d018ccf-plugins-config-volume") pod "payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8" (UID: "ed7a9c3a-35ec-40ba-8540-14136d018ccf") : failed to sync configmap cache: timed out waiting for the condition
Jul 30 03:04:41.836614 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:41.836567    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw" event={"ID":"9a77ddb9-9d12-47fb-a432-661b1c8cbaa9","Type":"ContainerStarted","Data":"334ba3816012d222b91141302e075c0c6830dceb1e9891e870219c188fc82d37"}
Jul 30 03:04:41.836791 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:41.836726    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"
Jul 30 03:04:41.884020 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:04:41.883985    2565 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-mt-adb22f94: failed to sync configmap cache: timed out waiting for the condition
Jul 30 03:04:41.884201 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:04:41.884079    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4152435-679b-4cba-9f41-f8692f90c305-plugins-config-volume podName:b4152435-679b-4cba-9f41-f8692f90c305 nodeName:}" failed. No retries permitted until 2026-07-30 03:04:42.384061134 +0000 UTC m=+2318.030844518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b4152435-679b-4cba-9f41-f8692f90c305-plugins-config-volume") pod "payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9" (UID: "b4152435-679b-4cba-9f41-f8692f90c305") : failed to sync configmap cache: timed out waiting for the condition
Jul 30 03:04:41.893142 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:41.893081    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw" podStartSLOduration=2.136998382 podStartE2EDuration="2.893064519s" totalImagesPullingTime="756.066137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:04:41.891671745 +0000 UTC m=+2317.538455163" watchObservedRunningTime="2026-07-30 03:04:41.893064519 +0000 UTC m=+2317.539848002"
Jul 30 03:04:41.914398 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:41.914364    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-adb22f94\""
Jul 30 03:04:42.196666 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:42.196575    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7a9c3a-35ec-40ba-8540-14136d018ccf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8\" (UID: \"ed7a9c3a-35ec-40ba-8540-14136d018ccf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8"
Jul 30 03:04:42.197261 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:42.197243    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7a9c3a-35ec-40ba-8540-14136d018ccf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8\" (UID: \"ed7a9c3a-35ec-40ba-8540-14136d018ccf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8"
Jul 30 03:04:42.298310 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:42.298281    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8"
Jul 30 03:04:42.399057 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:42.399017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4152435-679b-4cba-9f41-f8692f90c305-plugins-config-volume\") pod \"payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9\" (UID: \"b4152435-679b-4cba-9f41-f8692f90c305\") " pod="openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9"
Jul 30 03:04:42.399989 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:42.399965    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4152435-679b-4cba-9f41-f8692f90c305-plugins-config-volume\") pod \"payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9\" (UID: \"b4152435-679b-4cba-9f41-f8692f90c305\") " pod="openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9"
Jul 30 03:04:42.457365 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:42.457333    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8"]
Jul 30 03:04:42.459217 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:04:42.459192    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7a9c3a_35ec_40ba_8540_14136d018ccf.slice/crio-c68b9118d5c45c7713676f74112cfe85f9d5bc8505387dfd97642468bae85f2f WatchSource:0}: Error finding container c68b9118d5c45c7713676f74112cfe85f9d5bc8505387dfd97642468bae85f2f: Status 404 returned error can't find the container with id c68b9118d5c45c7713676f74112cfe85f9d5bc8505387dfd97642468bae85f2f
Jul 30 03:04:42.535333 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:42.535307    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9"
Jul 30 03:04:42.683599 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:42.683565    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9"]
Jul 30 03:04:42.684387 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:04:42.684359    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4152435_679b_4cba_9f41_f8692f90c305.slice/crio-7ca5693caa3dba3e508b80aa1be94166d9395b35947021db15487c603da41aea WatchSource:0}: Error finding container 7ca5693caa3dba3e508b80aa1be94166d9395b35947021db15487c603da41aea: Status 404 returned error can't find the container with id 7ca5693caa3dba3e508b80aa1be94166d9395b35947021db15487c603da41aea
Jul 30 03:04:42.844451 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:42.844411    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9" event={"ID":"b4152435-679b-4cba-9f41-f8692f90c305","Type":"ContainerStarted","Data":"c94702532df649ef7fb7fb3e98d0c5baf6bccaf00f0593f8b5043ffc89a8973f"}
Jul 30 03:04:42.844624 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:42.844478    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9"
Jul 30 03:04:42.844624 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:42.844506    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9" event={"ID":"b4152435-679b-4cba-9f41-f8692f90c305","Type":"ContainerStarted","Data":"7ca5693caa3dba3e508b80aa1be94166d9395b35947021db15487c603da41aea"}
Jul 30 03:04:42.846026 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:42.845998    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8" event={"ID":"ed7a9c3a-35ec-40ba-8540-14136d018ccf","Type":"ContainerStarted","Data":"e3d760b8097d071c6aea45106607430b620d685876cc089308a0cad465e69c9f"}
Jul 30 03:04:42.846183 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:42.846048    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8"
Jul 30 03:04:42.846183 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:42.846066    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8" event={"ID":"ed7a9c3a-35ec-40ba-8540-14136d018ccf","Type":"ContainerStarted","Data":"c68b9118d5c45c7713676f74112cfe85f9d5bc8505387dfd97642468bae85f2f"}
Jul 30 03:04:42.868795 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:42.868743    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9" podStartSLOduration=2.868726275 podStartE2EDuration="2.868726275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:04:42.864116055 +0000 UTC m=+2318.510899464" watchObservedRunningTime="2026-07-30 03:04:42.868726275 +0000 UTC m=+2318.515509708"
Jul 30 03:04:42.891469 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:42.891417    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8" podStartSLOduration=2.89140019 podStartE2EDuration="2.89140019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:04:42.887764767 +0000 UTC m=+2318.534548172" watchObservedRunningTime="2026-07-30 03:04:42.89140019 +0000 UTC m=+2318.538183595"
Jul 30 03:04:47.851743 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:47.851709    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"
Jul 30 03:04:53.855071 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:53.855034    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9"
Jul 30 03:04:53.856409 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:53.855124    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8"
Jul 30 03:04:53.885399 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:53.885364    2565 patch_prober.go:28] interesting pod/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": read tcp 10.134.0.2:49552->10.134.0.78:15021: read: connection reset by peer" start-of-body=
Jul 30 03:04:53.885543 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:53.885430    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck" podUID="594025e4-2c47-4fec-a6dc-c4ce28958b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": read tcp 10.134.0.2:49552->10.134.0.78:15021: read: connection reset by peer"
Jul 30 03:04:53.936822 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:53.936794    2565 generic.go:356] "Generic (PLEG): container finished" podID="594025e4-2c47-4fec-a6dc-c4ce28958b01" containerID="1a73f3dca6b98ba7836cae6b2ae12c0f2f5622ed9f47cde3ab12027d71f27640" exitCode=0
Jul 30 03:04:53.936982 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:53.936870    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck" event={"ID":"594025e4-2c47-4fec-a6dc-c4ce28958b01","Type":"ContainerDied","Data":"1a73f3dca6b98ba7836cae6b2ae12c0f2f5622ed9f47cde3ab12027d71f27640"}
Jul 30 03:04:53.937320 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:53.937305    2565 scope.go:117] "RemoveContainer" containerID="1a73f3dca6b98ba7836cae6b2ae12c0f2f5622ed9f47cde3ab12027d71f27640"
Jul 30 03:04:53.938493 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:53.938454    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:04:53.938585 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:53.938543    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:04:53.938628 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:53.938600    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:04:54.944455 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:54.944419    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck" event={"ID":"594025e4-2c47-4fec-a6dc-c4ce28958b01","Type":"ContainerStarted","Data":"fde350fe02c400dd7cce8edf054dce638a47aee3337e19f12672717e35db2efb"}
Jul 30 03:04:54.958615 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:54.958593    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:55.958924 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:55.958837    2565 patch_prober.go:28] interesting pod/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Jul 30 03:04:55.958924 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:55.958900    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck" podUID="594025e4-2c47-4fec-a6dc-c4ce28958b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Jul 30 03:04:56.958787 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:56.958754    2565 patch_prober.go:28] interesting pod/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Jul 30 03:04:56.958955 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:56.958826    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck" podUID="594025e4-2c47-4fec-a6dc-c4ce28958b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Jul 30 03:04:57.959854 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:57.959815    2565 patch_prober.go:28] interesting pod/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Jul 30 03:04:57.960364 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:57.959880    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck" podUID="594025e4-2c47-4fec-a6dc-c4ce28958b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Jul 30 03:04:58.959041 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:58.959001    2565 patch_prober.go:28] interesting pod/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Jul 30 03:04:58.959259 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:58.959077    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck" podUID="594025e4-2c47-4fec-a6dc-c4ce28958b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Jul 30 03:04:59.962673 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:59.962643    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:59.963046 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:59.962952    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:04:59.963660 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:04:59.963643    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:05:05.078650 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:05.078613    2565 scope.go:117] "RemoveContainer" containerID="4af8c98d6aee1ad5338622d25c7789ec215746c2ff93b2170eeb8d06380956e6"
Jul 30 03:05:05.088199 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:05.088154    2565 scope.go:117] "RemoveContainer" containerID="595d3e6f553ea772f9d85459db70d112042134829d22e6a5f58b6ad68ec2912f"
Jul 30 03:05:22.834557 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:22.834523    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb22f94-zkj9l"]
Jul 30 03:05:22.838355 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:22.838339    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb22f94-zkj9l"
Jul 30 03:05:22.842781 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:22.842760    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2npk7\""
Jul 30 03:05:22.854802 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:22.854778    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb22f94-zkj9l"]
Jul 30 03:05:22.890593 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:22.890558    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0231f05-f94e-407b-a32f-b1381f629504-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-adb22f94-zkj9l\" (UID: \"f0231f05-f94e-407b-a32f-b1381f629504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb22f94-zkj9l"
Jul 30 03:05:22.991152 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:22.991119    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0231f05-f94e-407b-a32f-b1381f629504-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-adb22f94-zkj9l\" (UID: \"f0231f05-f94e-407b-a32f-b1381f629504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb22f94-zkj9l"
Jul 30 03:05:22.991899 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:22.991875    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0231f05-f94e-407b-a32f-b1381f629504-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-adb22f94-zkj9l\" (UID: \"f0231f05-f94e-407b-a32f-b1381f629504\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb22f94-zkj9l"
Jul 30 03:05:23.148661 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:23.148570    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb22f94-zkj9l"
Jul 30 03:05:23.290287 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:23.290260    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb22f94-zkj9l"]
Jul 30 03:05:23.292665 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:05:23.292634    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0231f05_f94e_407b_a32f_b1381f629504.slice/crio-c758d88eefefa4214e34e5eea680ed21af6e38aac33676210049cb09caff82eb WatchSource:0}: Error finding container c758d88eefefa4214e34e5eea680ed21af6e38aac33676210049cb09caff82eb: Status 404 returned error can't find the container with id c758d88eefefa4214e34e5eea680ed21af6e38aac33676210049cb09caff82eb
Jul 30 03:05:24.123010 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:24.122974    2565 generic.go:356] "Generic (PLEG): container finished" podID="f0231f05-f94e-407b-a32f-b1381f629504" containerID="ee6a9cb062b09057ef3f6af069aeb2d7c2065868ce325b6b0f4ee10a84a0b44a" exitCode=0
Jul 30 03:05:24.123460 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:24.123028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb22f94-zkj9l" event={"ID":"f0231f05-f94e-407b-a32f-b1381f629504","Type":"ContainerDied","Data":"ee6a9cb062b09057ef3f6af069aeb2d7c2065868ce325b6b0f4ee10a84a0b44a"}
Jul 30 03:05:24.123460 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:24.123119    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb22f94-zkj9l" event={"ID":"f0231f05-f94e-407b-a32f-b1381f629504","Type":"ContainerStarted","Data":"c758d88eefefa4214e34e5eea680ed21af6e38aac33676210049cb09caff82eb"}
Jul 30 03:05:25.268483 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:25.268460    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb22f94-zkj9l"
Jul 30 03:05:25.414548 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:25.414456    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0231f05-f94e-407b-a32f-b1381f629504-maas-api-service-ca\") pod \"f0231f05-f94e-407b-a32f-b1381f629504\" (UID: \"f0231f05-f94e-407b-a32f-b1381f629504\") "
Jul 30 03:05:25.414773 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:25.414751    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0231f05-f94e-407b-a32f-b1381f629504-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f0231f05-f94e-407b-a32f-b1381f629504" (UID: "f0231f05-f94e-407b-a32f-b1381f629504"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:05:25.515858 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:25.515820    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0231f05-f94e-407b-a32f-b1381f629504-maas-api-service-ca\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:05:26.137389 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:26.137352    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb22f94-zkj9l"
Jul 30 03:05:26.137574 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:26.137390    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb22f94-zkj9l" event={"ID":"f0231f05-f94e-407b-a32f-b1381f629504","Type":"ContainerDied","Data":"c758d88eefefa4214e34e5eea680ed21af6e38aac33676210049cb09caff82eb"}
Jul 30 03:05:26.137574 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:26.137444    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c758d88eefefa4214e34e5eea680ed21af6e38aac33676210049cb09caff82eb"
Jul 30 03:05:38.278057 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.278011    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"]
Jul 30 03:05:38.278628 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.278289    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw" podUID="9a77ddb9-9d12-47fb-a432-661b1c8cbaa9" containerName="maas-api" containerID="cri-o://334ba3816012d222b91141302e075c0c6830dceb1e9891e870219c188fc82d37" gracePeriod=30
Jul 30 03:05:38.329685 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.329651    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9"]
Jul 30 03:05:38.329966 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.329917    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9" podUID="b4152435-679b-4cba-9f41-f8692f90c305" containerName="payload-processing" containerID="cri-o://c94702532df649ef7fb7fb3e98d0c5baf6bccaf00f0593f8b5043ffc89a8973f" gracePeriod=30
Jul 30 03:05:38.334896 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.334867    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8"]
Jul 30 03:05:38.335353 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.335300    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8" podUID="ed7a9c3a-35ec-40ba-8540-14136d018ccf" containerName="payload-pre-processing" containerID="cri-o://e3d760b8097d071c6aea45106607430b620d685876cc089308a0cad465e69c9f" gracePeriod=30
Jul 30 03:05:38.597410 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.597377    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"
Jul 30 03:05:38.613422 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.613387    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9"
Jul 30 03:05:38.643092 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.643065    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8"
Jul 30 03:05:38.748429 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.748398    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lcsl\" (UniqueName: \"kubernetes.io/projected/b4152435-679b-4cba-9f41-f8692f90c305-kube-api-access-7lcsl\") pod \"b4152435-679b-4cba-9f41-f8692f90c305\" (UID: \"b4152435-679b-4cba-9f41-f8692f90c305\") "
Jul 30 03:05:38.748638 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.748460    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c6jt\" (UniqueName: \"kubernetes.io/projected/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-kube-api-access-5c6jt\") pod \"9a77ddb9-9d12-47fb-a432-661b1c8cbaa9\" (UID: \"9a77ddb9-9d12-47fb-a432-661b1c8cbaa9\") "
Jul 30 03:05:38.748638 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.748490    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92bg6\" (UniqueName: \"kubernetes.io/projected/ed7a9c3a-35ec-40ba-8540-14136d018ccf-kube-api-access-92bg6\") pod \"ed7a9c3a-35ec-40ba-8540-14136d018ccf\" (UID: \"ed7a9c3a-35ec-40ba-8540-14136d018ccf\") "
Jul 30 03:05:38.748638 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.748522    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-openshift-service-ca\") pod \"9a77ddb9-9d12-47fb-a432-661b1c8cbaa9\" (UID: \"9a77ddb9-9d12-47fb-a432-661b1c8cbaa9\") "
Jul 30 03:05:38.748638 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.748549    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4152435-679b-4cba-9f41-f8692f90c305-plugins-config-volume\") pod \"b4152435-679b-4cba-9f41-f8692f90c305\" (UID: \"b4152435-679b-4cba-9f41-f8692f90c305\") "
Jul 30 03:05:38.748638 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.748570    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7a9c3a-35ec-40ba-8540-14136d018ccf-plugins-config-volume\") pod \"ed7a9c3a-35ec-40ba-8540-14136d018ccf\" (UID: \"ed7a9c3a-35ec-40ba-8540-14136d018ccf\") "
Jul 30 03:05:38.748638 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.748599    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-maas-api-tls\") pod \"9a77ddb9-9d12-47fb-a432-661b1c8cbaa9\" (UID: \"9a77ddb9-9d12-47fb-a432-661b1c8cbaa9\") "
Jul 30 03:05:38.748978 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.748922    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9a77ddb9-9d12-47fb-a432-661b1c8cbaa9" (UID: "9a77ddb9-9d12-47fb-a432-661b1c8cbaa9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:05:38.748978 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.748932    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4152435-679b-4cba-9f41-f8692f90c305-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b4152435-679b-4cba-9f41-f8692f90c305" (UID: "b4152435-679b-4cba-9f41-f8692f90c305"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:05:38.749211 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.749142    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7a9c3a-35ec-40ba-8540-14136d018ccf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed7a9c3a-35ec-40ba-8540-14136d018ccf" (UID: "ed7a9c3a-35ec-40ba-8540-14136d018ccf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:05:38.750922 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.750894    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-kube-api-access-5c6jt" (OuterVolumeSpecName: "kube-api-access-5c6jt") pod "9a77ddb9-9d12-47fb-a432-661b1c8cbaa9" (UID: "9a77ddb9-9d12-47fb-a432-661b1c8cbaa9"). InnerVolumeSpecName "kube-api-access-5c6jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:05:38.750922 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.750889    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9a77ddb9-9d12-47fb-a432-661b1c8cbaa9" (UID: "9a77ddb9-9d12-47fb-a432-661b1c8cbaa9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:05:38.751219 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.751189    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7a9c3a-35ec-40ba-8540-14136d018ccf-kube-api-access-92bg6" (OuterVolumeSpecName: "kube-api-access-92bg6") pod "ed7a9c3a-35ec-40ba-8540-14136d018ccf" (UID: "ed7a9c3a-35ec-40ba-8540-14136d018ccf"). InnerVolumeSpecName "kube-api-access-92bg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:05:38.751219 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.751193    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4152435-679b-4cba-9f41-f8692f90c305-kube-api-access-7lcsl" (OuterVolumeSpecName: "kube-api-access-7lcsl") pod "b4152435-679b-4cba-9f41-f8692f90c305" (UID: "b4152435-679b-4cba-9f41-f8692f90c305"). InnerVolumeSpecName "kube-api-access-7lcsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:05:38.850126 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.850090    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c6jt\" (UniqueName: \"kubernetes.io/projected/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-kube-api-access-5c6jt\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:05:38.850126 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.850122    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92bg6\" (UniqueName: \"kubernetes.io/projected/ed7a9c3a-35ec-40ba-8540-14136d018ccf-kube-api-access-92bg6\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:05:38.850126 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.850133    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-openshift-service-ca\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:05:38.850373 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.850161    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4152435-679b-4cba-9f41-f8692f90c305-plugins-config-volume\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:05:38.850373 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.850200    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7a9c3a-35ec-40ba-8540-14136d018ccf-plugins-config-volume\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:05:38.850373 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.850209    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9-maas-api-tls\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:05:38.850373 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:38.850217    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lcsl\" (UniqueName: \"kubernetes.io/projected/b4152435-679b-4cba-9f41-f8692f90c305-kube-api-access-7lcsl\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:05:39.217880 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.217790    2565 generic.go:356] "Generic (PLEG): container finished" podID="9a77ddb9-9d12-47fb-a432-661b1c8cbaa9" containerID="334ba3816012d222b91141302e075c0c6830dceb1e9891e870219c188fc82d37" exitCode=0
Jul 30 03:05:39.217880 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.217836    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"
Jul 30 03:05:39.218124 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.217879    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw" event={"ID":"9a77ddb9-9d12-47fb-a432-661b1c8cbaa9","Type":"ContainerDied","Data":"334ba3816012d222b91141302e075c0c6830dceb1e9891e870219c188fc82d37"}
Jul 30 03:05:39.218124 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.217928    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw" event={"ID":"9a77ddb9-9d12-47fb-a432-661b1c8cbaa9","Type":"ContainerDied","Data":"23730778247e0ace912a2ca44cf56f3a1b279384b4c1ca9e29d2d7294045d3e7"}
Jul 30 03:05:39.218124 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.217957    2565 scope.go:117] "RemoveContainer" containerID="334ba3816012d222b91141302e075c0c6830dceb1e9891e870219c188fc82d37"
Jul 30 03:05:39.219399 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.219375    2565 generic.go:356] "Generic (PLEG): container finished" podID="b4152435-679b-4cba-9f41-f8692f90c305" containerID="c94702532df649ef7fb7fb3e98d0c5baf6bccaf00f0593f8b5043ffc89a8973f" exitCode=0
Jul 30 03:05:39.219682 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.219415    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9"
Jul 30 03:05:39.219682 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.219415    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9" event={"ID":"b4152435-679b-4cba-9f41-f8692f90c305","Type":"ContainerDied","Data":"c94702532df649ef7fb7fb3e98d0c5baf6bccaf00f0593f8b5043ffc89a8973f"}
Jul 30 03:05:39.219682 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.219466    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9" event={"ID":"b4152435-679b-4cba-9f41-f8692f90c305","Type":"ContainerDied","Data":"7ca5693caa3dba3e508b80aa1be94166d9395b35947021db15487c603da41aea"}
Jul 30 03:05:39.221039 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.221016    2565 generic.go:356] "Generic (PLEG): container finished" podID="ed7a9c3a-35ec-40ba-8540-14136d018ccf" containerID="e3d760b8097d071c6aea45106607430b620d685876cc089308a0cad465e69c9f" exitCode=0
Jul 30 03:05:39.221157 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.221055    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8"
Jul 30 03:05:39.221157 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.221119    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8" event={"ID":"ed7a9c3a-35ec-40ba-8540-14136d018ccf","Type":"ContainerDied","Data":"e3d760b8097d071c6aea45106607430b620d685876cc089308a0cad465e69c9f"}
Jul 30 03:05:39.221507 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.221192    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8" event={"ID":"ed7a9c3a-35ec-40ba-8540-14136d018ccf","Type":"ContainerDied","Data":"c68b9118d5c45c7713676f74112cfe85f9d5bc8505387dfd97642468bae85f2f"}
Jul 30 03:05:39.228219 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.228200    2565 scope.go:117] "RemoveContainer" containerID="334ba3816012d222b91141302e075c0c6830dceb1e9891e870219c188fc82d37"
Jul 30 03:05:39.228495 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:05:39.228473    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"334ba3816012d222b91141302e075c0c6830dceb1e9891e870219c188fc82d37\": container with ID starting with 334ba3816012d222b91141302e075c0c6830dceb1e9891e870219c188fc82d37 not found: ID does not exist" containerID="334ba3816012d222b91141302e075c0c6830dceb1e9891e870219c188fc82d37"
Jul 30 03:05:39.228578 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.228517    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334ba3816012d222b91141302e075c0c6830dceb1e9891e870219c188fc82d37"} err="failed to get container status \"334ba3816012d222b91141302e075c0c6830dceb1e9891e870219c188fc82d37\": rpc error: code = NotFound desc = could not find container \"334ba3816012d222b91141302e075c0c6830dceb1e9891e870219c188fc82d37\": container with ID starting with 334ba3816012d222b91141302e075c0c6830dceb1e9891e870219c188fc82d37 not found: ID does not exist"
Jul 30 03:05:39.228578 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.228540    2565 scope.go:117] "RemoveContainer" containerID="c94702532df649ef7fb7fb3e98d0c5baf6bccaf00f0593f8b5043ffc89a8973f"
Jul 30 03:05:39.236620 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.236600    2565 scope.go:117] "RemoveContainer" containerID="c94702532df649ef7fb7fb3e98d0c5baf6bccaf00f0593f8b5043ffc89a8973f"
Jul 30 03:05:39.236865 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:05:39.236843    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c94702532df649ef7fb7fb3e98d0c5baf6bccaf00f0593f8b5043ffc89a8973f\": container with ID starting with c94702532df649ef7fb7fb3e98d0c5baf6bccaf00f0593f8b5043ffc89a8973f not found: ID does not exist" containerID="c94702532df649ef7fb7fb3e98d0c5baf6bccaf00f0593f8b5043ffc89a8973f"
Jul 30 03:05:39.236919 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.236871    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c94702532df649ef7fb7fb3e98d0c5baf6bccaf00f0593f8b5043ffc89a8973f"} err="failed to get container status \"c94702532df649ef7fb7fb3e98d0c5baf6bccaf00f0593f8b5043ffc89a8973f\": rpc error: code = NotFound desc = could not find container \"c94702532df649ef7fb7fb3e98d0c5baf6bccaf00f0593f8b5043ffc89a8973f\": container with ID starting with c94702532df649ef7fb7fb3e98d0c5baf6bccaf00f0593f8b5043ffc89a8973f not found: ID does not exist"
Jul 30 03:05:39.236919 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.236886    2565 scope.go:117] "RemoveContainer" containerID="e3d760b8097d071c6aea45106607430b620d685876cc089308a0cad465e69c9f"
Jul 30 03:05:39.244064 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.244037    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9"]
Jul 30 03:05:39.245877 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.245860    2565 scope.go:117] "RemoveContainer" containerID="e3d760b8097d071c6aea45106607430b620d685876cc089308a0cad465e69c9f"
Jul 30 03:05:39.246141 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:05:39.246120    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d760b8097d071c6aea45106607430b620d685876cc089308a0cad465e69c9f\": container with ID starting with e3d760b8097d071c6aea45106607430b620d685876cc089308a0cad465e69c9f not found: ID does not exist" containerID="e3d760b8097d071c6aea45106607430b620d685876cc089308a0cad465e69c9f"
Jul 30 03:05:39.246227 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.246148    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d760b8097d071c6aea45106607430b620d685876cc089308a0cad465e69c9f"} err="failed to get container status \"e3d760b8097d071c6aea45106607430b620d685876cc089308a0cad465e69c9f\": rpc error: code = NotFound desc = could not find container \"e3d760b8097d071c6aea45106607430b620d685876cc089308a0cad465e69c9f\": container with ID starting with e3d760b8097d071c6aea45106607430b620d685876cc089308a0cad465e69c9f not found: ID does not exist"
Jul 30 03:05:39.246524 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.246508    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-adb22f94-77c9c5b49d-z5tw9"]
Jul 30 03:05:39.257325 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.257299    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"]
Jul 30 03:05:39.261877 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.261857    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-adb22f94-979695fd7-q98pw"]
Jul 30 03:05:39.273410 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.273383    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8"]
Jul 30 03:05:39.276668 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:39.276647    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-adb22f94-97465b8bd-mxqq8"]
Jul 30 03:05:40.800561 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:40.800524    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a77ddb9-9d12-47fb-a432-661b1c8cbaa9" path="/var/lib/kubelet/pods/9a77ddb9-9d12-47fb-a432-661b1c8cbaa9/volumes"
Jul 30 03:05:40.800930 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:40.800863    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4152435-679b-4cba-9f41-f8692f90c305" path="/var/lib/kubelet/pods/b4152435-679b-4cba-9f41-f8692f90c305/volumes"
Jul 30 03:05:40.801199 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:40.801184    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed7a9c3a-35ec-40ba-8540-14136d018ccf" path="/var/lib/kubelet/pods/ed7a9c3a-35ec-40ba-8540-14136d018ccf/volumes"
Jul 30 03:05:42.896403 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:42.896365    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb22f94-zkj9l"]
Jul 30 03:05:42.899858 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:42.899833    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb22f94-zkj9l"]
Jul 30 03:05:44.801289 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:05:44.801252    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0231f05-f94e-407b-a32f-b1381f629504" path="/var/lib/kubelet/pods/f0231f05-f94e-407b-a32f-b1381f629504/volumes"
Jul 30 03:06:05.117704 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:05.117668    2565 scope.go:117] "RemoveContainer" containerID="0fd3d9d41f57aba357a4d18f25ac623498f73db9df121a9591176876c5f3a65f"
Jul 30 03:06:05.127096 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:05.127074    2565 scope.go:117] "RemoveContainer" containerID="4d8443a1a18ffe3142a4180221bef139e3d21d675a773df7971e0b474afcd8d4"
Jul 30 03:06:05.135421 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:05.135393    2565 scope.go:117] "RemoveContainer" containerID="232c140f8ffc80e492a6eb7106e758859e4c93718b5a812451a0a616b2a8188f"
Jul 30 03:06:33.833806 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:33.833720    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"]
Jul 30 03:06:33.834416 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:33.834000    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck" podUID="594025e4-2c47-4fec-a6dc-c4ce28958b01" containerName="istio-proxy" containerID="cri-o://fde350fe02c400dd7cce8edf054dce638a47aee3337e19f12672717e35db2efb" gracePeriod=30
Jul 30 03:06:39.085979 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.085954    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:06:39.196633 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.196508    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gs4d\" (UniqueName: \"kubernetes.io/projected/594025e4-2c47-4fec-a6dc-c4ce28958b01-kube-api-access-5gs4d\") pod \"594025e4-2c47-4fec-a6dc-c4ce28958b01\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") "
Jul 30 03:06:39.196633 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.196568    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/594025e4-2c47-4fec-a6dc-c4ce28958b01-istiod-ca-cert\") pod \"594025e4-2c47-4fec-a6dc-c4ce28958b01\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") "
Jul 30 03:06:39.196633 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.196615    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-workload-certs\") pod \"594025e4-2c47-4fec-a6dc-c4ce28958b01\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") "
Jul 30 03:06:39.196923 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.196660    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-envoy\") pod \"594025e4-2c47-4fec-a6dc-c4ce28958b01\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") "
Jul 30 03:06:39.196923 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.196728    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-credential-socket\") pod \"594025e4-2c47-4fec-a6dc-c4ce28958b01\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") "
Jul 30 03:06:39.196923 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.196755    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-token\") pod \"594025e4-2c47-4fec-a6dc-c4ce28958b01\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") "
Jul 30 03:06:39.196923 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.196795    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-podinfo\") pod \"594025e4-2c47-4fec-a6dc-c4ce28958b01\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") "
Jul 30 03:06:39.196923 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.196824    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-workload-socket\") pod \"594025e4-2c47-4fec-a6dc-c4ce28958b01\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") "
Jul 30 03:06:39.196923 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.196867    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-data\") pod \"594025e4-2c47-4fec-a6dc-c4ce28958b01\" (UID: \"594025e4-2c47-4fec-a6dc-c4ce28958b01\") "
Jul 30 03:06:39.197372 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.196931    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "594025e4-2c47-4fec-a6dc-c4ce28958b01" (UID: "594025e4-2c47-4fec-a6dc-c4ce28958b01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:06:39.197372 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.196968    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594025e4-2c47-4fec-a6dc-c4ce28958b01-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "594025e4-2c47-4fec-a6dc-c4ce28958b01" (UID: "594025e4-2c47-4fec-a6dc-c4ce28958b01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:06:39.197372 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.197132    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "594025e4-2c47-4fec-a6dc-c4ce28958b01" (UID: "594025e4-2c47-4fec-a6dc-c4ce28958b01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:06:39.197372 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.197311    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "594025e4-2c47-4fec-a6dc-c4ce28958b01" (UID: "594025e4-2c47-4fec-a6dc-c4ce28958b01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:06:39.197372 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.197320    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/594025e4-2c47-4fec-a6dc-c4ce28958b01-istiod-ca-cert\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:06:39.197372 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.197343    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-workload-certs\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:06:39.197372 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.197355    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-credential-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:06:39.198008 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.197981    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-data" (OuterVolumeSpecName: "istio-data") pod "594025e4-2c47-4fec-a6dc-c4ce28958b01" (UID: "594025e4-2c47-4fec-a6dc-c4ce28958b01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:06:39.199497 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.199467    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-token" (OuterVolumeSpecName: "istio-token") pod "594025e4-2c47-4fec-a6dc-c4ce28958b01" (UID: "594025e4-2c47-4fec-a6dc-c4ce28958b01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:06:39.199497 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.199482    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "594025e4-2c47-4fec-a6dc-c4ce28958b01" (UID: "594025e4-2c47-4fec-a6dc-c4ce28958b01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 30 03:06:39.199678 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.199624    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "594025e4-2c47-4fec-a6dc-c4ce28958b01" (UID: "594025e4-2c47-4fec-a6dc-c4ce28958b01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:06:39.199767 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.199746    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594025e4-2c47-4fec-a6dc-c4ce28958b01-kube-api-access-5gs4d" (OuterVolumeSpecName: "kube-api-access-5gs4d") pod "594025e4-2c47-4fec-a6dc-c4ce28958b01" (UID: "594025e4-2c47-4fec-a6dc-c4ce28958b01"). InnerVolumeSpecName "kube-api-access-5gs4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:06:39.298574 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.298539    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-envoy\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:06:39.298574 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.298567    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-token\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:06:39.298574 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.298576    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-podinfo\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:06:39.298574 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.298584    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-workload-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:06:39.298837 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.298592    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/594025e4-2c47-4fec-a6dc-c4ce28958b01-istio-data\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:06:39.298837 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.298602    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gs4d\" (UniqueName: \"kubernetes.io/projected/594025e4-2c47-4fec-a6dc-c4ce28958b01-kube-api-access-5gs4d\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:06:39.559730 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.559695    2565 generic.go:356] "Generic (PLEG): container finished" podID="594025e4-2c47-4fec-a6dc-c4ce28958b01" containerID="fde350fe02c400dd7cce8edf054dce638a47aee3337e19f12672717e35db2efb" exitCode=0
Jul 30 03:06:39.559921 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.559755    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"
Jul 30 03:06:39.559921 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.559775    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck" event={"ID":"594025e4-2c47-4fec-a6dc-c4ce28958b01","Type":"ContainerDied","Data":"fde350fe02c400dd7cce8edf054dce638a47aee3337e19f12672717e35db2efb"}
Jul 30 03:06:39.559921 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.559827    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck" event={"ID":"594025e4-2c47-4fec-a6dc-c4ce28958b01","Type":"ContainerDied","Data":"c9ff6bd4e81f562f888216981315d7e432ea6ad1758c6aaee46c25f363537afe"}
Jul 30 03:06:39.559921 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.559848    2565 scope.go:117] "RemoveContainer" containerID="fde350fe02c400dd7cce8edf054dce638a47aee3337e19f12672717e35db2efb"
Jul 30 03:06:39.570073 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.570057    2565 scope.go:117] "RemoveContainer" containerID="1a73f3dca6b98ba7836cae6b2ae12c0f2f5622ed9f47cde3ab12027d71f27640"
Jul 30 03:06:39.578435 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.578412    2565 scope.go:117] "RemoveContainer" containerID="fde350fe02c400dd7cce8edf054dce638a47aee3337e19f12672717e35db2efb"
Jul 30 03:06:39.578727 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:06:39.578703    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde350fe02c400dd7cce8edf054dce638a47aee3337e19f12672717e35db2efb\": container with ID starting with fde350fe02c400dd7cce8edf054dce638a47aee3337e19f12672717e35db2efb not found: ID does not exist" containerID="fde350fe02c400dd7cce8edf054dce638a47aee3337e19f12672717e35db2efb"
Jul 30 03:06:39.578781 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.578739    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde350fe02c400dd7cce8edf054dce638a47aee3337e19f12672717e35db2efb"} err="failed to get container status \"fde350fe02c400dd7cce8edf054dce638a47aee3337e19f12672717e35db2efb\": rpc error: code = NotFound desc = could not find container \"fde350fe02c400dd7cce8edf054dce638a47aee3337e19f12672717e35db2efb\": container with ID starting with fde350fe02c400dd7cce8edf054dce638a47aee3337e19f12672717e35db2efb not found: ID does not exist"
Jul 30 03:06:39.578781 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.578763    2565 scope.go:117] "RemoveContainer" containerID="1a73f3dca6b98ba7836cae6b2ae12c0f2f5622ed9f47cde3ab12027d71f27640"
Jul 30 03:06:39.579023 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:06:39.579006    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a73f3dca6b98ba7836cae6b2ae12c0f2f5622ed9f47cde3ab12027d71f27640\": container with ID starting with 1a73f3dca6b98ba7836cae6b2ae12c0f2f5622ed9f47cde3ab12027d71f27640 not found: ID does not exist" containerID="1a73f3dca6b98ba7836cae6b2ae12c0f2f5622ed9f47cde3ab12027d71f27640"
Jul 30 03:06:39.579082 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.579031    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a73f3dca6b98ba7836cae6b2ae12c0f2f5622ed9f47cde3ab12027d71f27640"} err="failed to get container status \"1a73f3dca6b98ba7836cae6b2ae12c0f2f5622ed9f47cde3ab12027d71f27640\": rpc error: code = NotFound desc = could not find container \"1a73f3dca6b98ba7836cae6b2ae12c0f2f5622ed9f47cde3ab12027d71f27640\": container with ID starting with 1a73f3dca6b98ba7836cae6b2ae12c0f2f5622ed9f47cde3ab12027d71f27640 not found: ID does not exist"
Jul 30 03:06:39.583359 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.582942    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"]
Jul 30 03:06:39.585421 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:39.585401    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-adb22f94-openshift-default-8ddf76dd-8v9ck"]
Jul 30 03:06:40.800995 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:06:40.800963    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="594025e4-2c47-4fec-a6dc-c4ce28958b01" path="/var/lib/kubelet/pods/594025e4-2c47-4fec-a6dc-c4ce28958b01/volumes"
Jul 30 03:07:28.606577 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.606544    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"]
Jul 30 03:07:28.606996 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.606944    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="594025e4-2c47-4fec-a6dc-c4ce28958b01" containerName="istio-proxy"
Jul 30 03:07:28.606996 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.606956    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="594025e4-2c47-4fec-a6dc-c4ce28958b01" containerName="istio-proxy"
Jul 30 03:07:28.606996 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.606965    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a77ddb9-9d12-47fb-a432-661b1c8cbaa9" containerName="maas-api"
Jul 30 03:07:28.606996 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.606971    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a77ddb9-9d12-47fb-a432-661b1c8cbaa9" containerName="maas-api"
Jul 30 03:07:28.606996 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.606983    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4152435-679b-4cba-9f41-f8692f90c305" containerName="payload-processing"
Jul 30 03:07:28.606996 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.606989    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4152435-679b-4cba-9f41-f8692f90c305" containerName="payload-processing"
Jul 30 03:07:28.606996 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.606997    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="594025e4-2c47-4fec-a6dc-c4ce28958b01" containerName="istio-proxy"
Jul 30 03:07:28.606996 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.607002    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="594025e4-2c47-4fec-a6dc-c4ce28958b01" containerName="istio-proxy"
Jul 30 03:07:28.607306 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.607015    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0231f05-f94e-407b-a32f-b1381f629504" containerName="revoke-keys"
Jul 30 03:07:28.607306 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.607020    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0231f05-f94e-407b-a32f-b1381f629504" containerName="revoke-keys"
Jul 30 03:07:28.607306 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.607028    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed7a9c3a-35ec-40ba-8540-14136d018ccf" containerName="payload-pre-processing"
Jul 30 03:07:28.607306 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.607033    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed7a9c3a-35ec-40ba-8540-14136d018ccf" containerName="payload-pre-processing"
Jul 30 03:07:28.607306 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.607087    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0231f05-f94e-407b-a32f-b1381f629504" containerName="revoke-keys"
Jul 30 03:07:28.607306 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.607095    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed7a9c3a-35ec-40ba-8540-14136d018ccf" containerName="payload-pre-processing"
Jul 30 03:07:28.607306 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.607102    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="594025e4-2c47-4fec-a6dc-c4ce28958b01" containerName="istio-proxy"
Jul 30 03:07:28.607306 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.607109    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4152435-679b-4cba-9f41-f8692f90c305" containerName="payload-processing"
Jul 30 03:07:28.607306 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.607116    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="594025e4-2c47-4fec-a6dc-c4ce28958b01" containerName="istio-proxy"
Jul 30 03:07:28.607306 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.607122    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a77ddb9-9d12-47fb-a432-661b1c8cbaa9" containerName="maas-api"
Jul 30 03:07:28.610139 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.610123    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.612980 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.612957    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3b8039-openshift-default-dockercfg-j8glr\""
Jul 30 03:07:28.622815 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.622792    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"]
Jul 30 03:07:28.656807 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.656771    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-workload-socket\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.656807 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.656806    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-credential-socket\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.656986 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.656834    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-data\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.656986 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.656893    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-token\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.656986 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.656974    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-workload-certs\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.657098 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.656992    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istiod-ca-cert\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.657098 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.657011    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-podinfo\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.657098 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.657032    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc5pm\" (UniqueName: \"kubernetes.io/projected/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-kube-api-access-sc5pm\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.657098 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.657053    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-envoy\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.758124 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.758091    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-workload-certs\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.758124 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.758132    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istiod-ca-cert\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.758420 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.758162    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-podinfo\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.758420 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.758221    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc5pm\" (UniqueName: \"kubernetes.io/projected/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-kube-api-access-sc5pm\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.758420 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.758250    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-envoy\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.758420 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.758302    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-workload-socket\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.758420 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.758333    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-credential-socket\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.758714 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.758574    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-workload-certs\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.758714 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.758571    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-data\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.758714 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.758643    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-token\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.758714 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.758701    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-credential-socket\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.758905 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.758837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-workload-socket\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.758905 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.758850    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-data\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.759209 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.758967    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istiod-ca-cert\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.760816 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.760794    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-envoy\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.761044 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.761021    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-podinfo\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.766740 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.766716    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-token\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.766916 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.766892    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc5pm\" (UniqueName: \"kubernetes.io/projected/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-kube-api-access-sc5pm\") pod \"e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") " pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:28.921796 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:28.921698    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:29.065270 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:29.065245    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"]
Jul 30 03:07:29.066255 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:07:29.066229    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83eff9b_ae72_4f2c_8c25_bb333b3713ee.slice/crio-6a52a9f5511d278ba12a9dada657f3b3674354ab9bab36ee9b10e2943dc9c19b WatchSource:0}: Error finding container 6a52a9f5511d278ba12a9dada657f3b3674354ab9bab36ee9b10e2943dc9c19b: Status 404 returned error can't find the container with id 6a52a9f5511d278ba12a9dada657f3b3674354ab9bab36ee9b10e2943dc9c19b
Jul 30 03:07:29.068840 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:29.068800    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:07:29.068933 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:29.068887    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:07:29.068993 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:29.068936    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:07:29.839909 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:29.839868    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr" event={"ID":"c83eff9b-ae72-4f2c-8c25-bb333b3713ee","Type":"ContainerStarted","Data":"b619ab6572cf26c6d908d76554898560f06d37e15c27443ab12a125a82ac192e"}
Jul 30 03:07:29.840305 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:29.839925    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr" event={"ID":"c83eff9b-ae72-4f2c-8c25-bb333b3713ee","Type":"ContainerStarted","Data":"6a52a9f5511d278ba12a9dada657f3b3674354ab9bab36ee9b10e2943dc9c19b"}
Jul 30 03:07:29.863694 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:29.863644    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr" podStartSLOduration=1.863584699 podStartE2EDuration="1.863584699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:07:29.860205219 +0000 UTC m=+2485.506988627" watchObservedRunningTime="2026-07-30 03:07:29.863584699 +0000 UTC m=+2485.510368106"
Jul 30 03:07:29.922129 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:29.922080    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:30.927305 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:30.927276    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:31.848732 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:31.848693    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"]
Jul 30 03:07:31.852577 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:31.852555    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:31.852731 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:31.852607    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:31.852731 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:31.852660    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"
Jul 30 03:07:31.855231 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:31.855212    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3b8039-serving-cert\""
Jul 30 03:07:31.865833 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:31.865803    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"]
Jul 30 03:07:31.991922 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:31.991883    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cb47e7-ee70-4010-b03e-afb498527c2c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw\" (UID: \"56cb47e7-ee70-4010-b03e-afb498527c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"
Jul 30 03:07:31.992339 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:31.991951    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzl8\" (UniqueName: \"kubernetes.io/projected/56cb47e7-ee70-4010-b03e-afb498527c2c-kube-api-access-6pzl8\") pod \"maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw\" (UID: \"56cb47e7-ee70-4010-b03e-afb498527c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"
Jul 30 03:07:31.992339 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:31.992000    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56cb47e7-ee70-4010-b03e-afb498527c2c-maas-api-tls\") pod \"maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw\" (UID: \"56cb47e7-ee70-4010-b03e-afb498527c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"
Jul 30 03:07:32.092961 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.092925    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56cb47e7-ee70-4010-b03e-afb498527c2c-maas-api-tls\") pod \"maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw\" (UID: \"56cb47e7-ee70-4010-b03e-afb498527c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"
Jul 30 03:07:32.093209 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.093016    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cb47e7-ee70-4010-b03e-afb498527c2c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw\" (UID: \"56cb47e7-ee70-4010-b03e-afb498527c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"
Jul 30 03:07:32.093209 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.093046    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzl8\" (UniqueName: \"kubernetes.io/projected/56cb47e7-ee70-4010-b03e-afb498527c2c-kube-api-access-6pzl8\") pod \"maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw\" (UID: \"56cb47e7-ee70-4010-b03e-afb498527c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"
Jul 30 03:07:32.093750 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.093690    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cb47e7-ee70-4010-b03e-afb498527c2c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw\" (UID: \"56cb47e7-ee70-4010-b03e-afb498527c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"
Jul 30 03:07:32.095579 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.095552    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56cb47e7-ee70-4010-b03e-afb498527c2c-maas-api-tls\") pod \"maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw\" (UID: \"56cb47e7-ee70-4010-b03e-afb498527c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"
Jul 30 03:07:32.103121 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.103065    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzl8\" (UniqueName: \"kubernetes.io/projected/56cb47e7-ee70-4010-b03e-afb498527c2c-kube-api-access-6pzl8\") pod \"maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw\" (UID: \"56cb47e7-ee70-4010-b03e-afb498527c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"
Jul 30 03:07:32.164092 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.164057    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"
Jul 30 03:07:32.306600 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.306563    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"]
Jul 30 03:07:32.307209 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:07:32.307178    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56cb47e7_ee70_4010_b03e_afb498527c2c.slice/crio-f28658ddc569cdf0fa891e511799c1407c90de30fb2f6cb3cc6e25091ca88963 WatchSource:0}: Error finding container f28658ddc569cdf0fa891e511799c1407c90de30fb2f6cb3cc6e25091ca88963: Status 404 returned error can't find the container with id f28658ddc569cdf0fa891e511799c1407c90de30fb2f6cb3cc6e25091ca88963
Jul 30 03:07:32.550443 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.550404    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4"]
Jul 30 03:07:32.555156 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.555138    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4"
Jul 30 03:07:32.557932 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.557910    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3b8039\""
Jul 30 03:07:32.558053 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.557979    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3b8039-dockercfg-kwm5z\""
Jul 30 03:07:32.568511 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.568486    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4"]
Jul 30 03:07:32.578803 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.578772    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t"]
Jul 30 03:07:32.582955 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.582932    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t"
Jul 30 03:07:32.597457 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.597433    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t"]
Jul 30 03:07:32.700114 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.700077    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqcfq\" (UniqueName: \"kubernetes.io/projected/72d9ae38-8e67-43a0-924d-2dd017b7fc58-kube-api-access-hqcfq\") pod \"payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t\" (UID: \"72d9ae38-8e67-43a0-924d-2dd017b7fc58\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t"
Jul 30 03:07:32.700330 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.700132    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv7jl\" (UniqueName: \"kubernetes.io/projected/ba1c1018-4261-45fc-b5a4-3c4eabb35768-kube-api-access-vv7jl\") pod \"payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4\" (UID: \"ba1c1018-4261-45fc-b5a4-3c4eabb35768\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4"
Jul 30 03:07:32.700330 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.700178    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d9ae38-8e67-43a0-924d-2dd017b7fc58-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t\" (UID: \"72d9ae38-8e67-43a0-924d-2dd017b7fc58\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t"
Jul 30 03:07:32.700330 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.700310    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba1c1018-4261-45fc-b5a4-3c4eabb35768-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4\" (UID: \"ba1c1018-4261-45fc-b5a4-3c4eabb35768\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4"
Jul 30 03:07:32.801466 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.801377    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba1c1018-4261-45fc-b5a4-3c4eabb35768-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4\" (UID: \"ba1c1018-4261-45fc-b5a4-3c4eabb35768\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4"
Jul 30 03:07:32.801466 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.801439    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqcfq\" (UniqueName: \"kubernetes.io/projected/72d9ae38-8e67-43a0-924d-2dd017b7fc58-kube-api-access-hqcfq\") pod \"payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t\" (UID: \"72d9ae38-8e67-43a0-924d-2dd017b7fc58\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t"
Jul 30 03:07:32.801707 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.801489    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv7jl\" (UniqueName: \"kubernetes.io/projected/ba1c1018-4261-45fc-b5a4-3c4eabb35768-kube-api-access-vv7jl\") pod \"payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4\" (UID: \"ba1c1018-4261-45fc-b5a4-3c4eabb35768\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4"
Jul 30 03:07:32.801707 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.801525    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d9ae38-8e67-43a0-924d-2dd017b7fc58-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t\" (UID: \"72d9ae38-8e67-43a0-924d-2dd017b7fc58\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t"
Jul 30 03:07:32.802084 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.802065    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba1c1018-4261-45fc-b5a4-3c4eabb35768-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4\" (UID: \"ba1c1018-4261-45fc-b5a4-3c4eabb35768\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4"
Jul 30 03:07:32.802491 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.802470    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d9ae38-8e67-43a0-924d-2dd017b7fc58-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t\" (UID: \"72d9ae38-8e67-43a0-924d-2dd017b7fc58\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t"
Jul 30 03:07:32.809652 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.809625    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv7jl\" (UniqueName: \"kubernetes.io/projected/ba1c1018-4261-45fc-b5a4-3c4eabb35768-kube-api-access-vv7jl\") pod \"payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4\" (UID: \"ba1c1018-4261-45fc-b5a4-3c4eabb35768\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4"
Jul 30 03:07:32.809771 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.809661    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqcfq\" (UniqueName: \"kubernetes.io/projected/72d9ae38-8e67-43a0-924d-2dd017b7fc58-kube-api-access-hqcfq\") pod \"payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t\" (UID: \"72d9ae38-8e67-43a0-924d-2dd017b7fc58\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t"
Jul 30 03:07:32.857383 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.857339    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw" event={"ID":"56cb47e7-ee70-4010-b03e-afb498527c2c","Type":"ContainerStarted","Data":"e9dc7080db493d24f0966ab15699010b2e25b56f3f5042956d7c6af748ceaca1"}
Jul 30 03:07:32.857591 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.857402    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw" event={"ID":"56cb47e7-ee70-4010-b03e-afb498527c2c","Type":"ContainerStarted","Data":"f28658ddc569cdf0fa891e511799c1407c90de30fb2f6cb3cc6e25091ca88963"}
Jul 30 03:07:32.866574 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.866547    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4"
Jul 30 03:07:32.893827 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:32.893788    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t"
Jul 30 03:07:33.036289 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:33.036231    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw" podStartSLOduration=1.629500267 podStartE2EDuration="2.036156061s" totalImagesPullingTime="406.655794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:07:32.876795177 +0000 UTC m=+2488.523578583" watchObservedRunningTime="2026-07-30 03:07:33.036156061 +0000 UTC m=+2488.682939468"
Jul 30 03:07:33.036683 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:33.036654    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4"]
Jul 30 03:07:33.037742 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:07:33.037700    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1c1018_4261_45fc_b5a4_3c4eabb35768.slice/crio-564c3a06f2b7c7d43a6931a92c8dd9827f5e70b2e5fc1a88d5d347e9a14cb3a5 WatchSource:0}: Error finding container 564c3a06f2b7c7d43a6931a92c8dd9827f5e70b2e5fc1a88d5d347e9a14cb3a5: Status 404 returned error can't find the container with id 564c3a06f2b7c7d43a6931a92c8dd9827f5e70b2e5fc1a88d5d347e9a14cb3a5
Jul 30 03:07:33.065370 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:33.065342    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t"]
Jul 30 03:07:33.866093 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:33.866056    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t" event={"ID":"72d9ae38-8e67-43a0-924d-2dd017b7fc58","Type":"ContainerStarted","Data":"7649e46431a5b0ff38ec40de3c571253e9960b66fdaeb0f031dbbf7bc6000022"}
Jul 30 03:07:33.866323 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:33.866126    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t"
Jul 30 03:07:33.866323 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:33.866153    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t" event={"ID":"72d9ae38-8e67-43a0-924d-2dd017b7fc58","Type":"ContainerStarted","Data":"ad0c209133d7cdbeb36d6ee44e70220a89893ce0747d6c6247554cf8518e9ce4"}
Jul 30 03:07:33.867534 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:33.867508    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4" event={"ID":"ba1c1018-4261-45fc-b5a4-3c4eabb35768","Type":"ContainerStarted","Data":"4f6a17d42156eddff13a3e59ac4b04facf5a2ea61f6a87c00f8cebed73ed538d"}
Jul 30 03:07:33.867638 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:33.867560    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4" event={"ID":"ba1c1018-4261-45fc-b5a4-3c4eabb35768","Type":"ContainerStarted","Data":"564c3a06f2b7c7d43a6931a92c8dd9827f5e70b2e5fc1a88d5d347e9a14cb3a5"}
Jul 30 03:07:33.867694 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:33.867663    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4"
Jul 30 03:07:33.867811 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:33.867798    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"
Jul 30 03:07:33.884475 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:33.884412    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t" podStartSLOduration=1.8843939939999999 podStartE2EDuration="1.884393994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:07:33.881185807 +0000 UTC m=+2489.527969208" watchObservedRunningTime="2026-07-30 03:07:33.884393994 +0000 UTC m=+2489.531177405"
Jul 30 03:07:33.897917 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:33.897867    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4" podStartSLOduration=1.897850446 podStartE2EDuration="1.897850446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:07:33.897504422 +0000 UTC m=+2489.544287827" watchObservedRunningTime="2026-07-30 03:07:33.897850446 +0000 UTC m=+2489.544633852"
Jul 30 03:07:36.936412 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:36.936359    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"]
Jul 30 03:07:36.940275 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:36.940252    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:36.943202 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:36.943182    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cb71ac-openshift-default-dockercfg-r5hbc\""
Jul 30 03:07:36.954150 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:36.954125    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"]
Jul 30 03:07:37.042651 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.042611    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-envoy\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.042651 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.042654    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-workload-certs\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.042923 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.042677    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-data\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.042923 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.042696    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istiod-ca-cert\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.042923 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.042769    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-credential-socket\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.042923 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.042898    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr47q\" (UniqueName: \"kubernetes.io/projected/ab96c315-6328-41ed-8ccd-eecfd828f6d9-kube-api-access-tr47q\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.043144 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.042940    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-podinfo\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.043144 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.042967    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-token\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.043144 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.043038    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-workload-socket\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.144013 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.143969    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr47q\" (UniqueName: \"kubernetes.io/projected/ab96c315-6328-41ed-8ccd-eecfd828f6d9-kube-api-access-tr47q\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.144013 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.144016    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-podinfo\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.144267 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.144033    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-token\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.144267 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.144083    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-workload-socket\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.144267 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.144138    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-envoy\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.144267 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.144198    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-workload-certs\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.144267 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.144224    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-data\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.144267 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.144249    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istiod-ca-cert\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.144574 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.144361    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-credential-socket\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.144665 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.144648    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-workload-socket\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.144722 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.144699    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-data\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.144768 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.144753    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-credential-socket\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.144804 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.144773    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-workload-certs\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.145302 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.145274    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istiod-ca-cert\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.146795 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.146776    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-envoy\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.146915 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.146900    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-podinfo\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.156142 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.156113    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-token\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.156386 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.156363    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr47q\" (UniqueName: \"kubernetes.io/projected/ab96c315-6328-41ed-8ccd-eecfd828f6d9-kube-api-access-tr47q\") pod \"e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") " pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.251352 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.251252    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:37.402456 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.402429    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"]
Jul 30 03:07:37.403317 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:07:37.403295    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab96c315_6328_41ed_8ccd_eecfd828f6d9.slice/crio-64d79e464865df552f201d2379b06378e5e2d2e441788ec089c053853a575031 WatchSource:0}: Error finding container 64d79e464865df552f201d2379b06378e5e2d2e441788ec089c053853a575031: Status 404 returned error can't find the container with id 64d79e464865df552f201d2379b06378e5e2d2e441788ec089c053853a575031
Jul 30 03:07:37.405516 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.405478    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:07:37.405626 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.405560    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:07:37.405626 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.405605    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:07:37.895868 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.895833    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84" event={"ID":"ab96c315-6328-41ed-8ccd-eecfd828f6d9","Type":"ContainerStarted","Data":"b725ee2bb39f83c5dd257f5c8b75b3314f380d5ac9354e4f5de68264d6662591"}
Jul 30 03:07:37.896053 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.895881    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84" event={"ID":"ab96c315-6328-41ed-8ccd-eecfd828f6d9","Type":"ContainerStarted","Data":"64d79e464865df552f201d2379b06378e5e2d2e441788ec089c053853a575031"}
Jul 30 03:07:37.918431 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:37.918379    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84" podStartSLOduration=1.918363035 podStartE2EDuration="1.918363035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:07:37.915908754 +0000 UTC m=+2493.562692159" watchObservedRunningTime="2026-07-30 03:07:37.918363035 +0000 UTC m=+2493.565146439"
Jul 30 03:07:38.251769 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:38.251673    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:38.257089 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:38.257053    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:38.901619 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:38.901591    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:38.902680 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:38.902655    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:39.882525 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:39.882495    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"
Jul 30 03:07:39.993377 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:39.993341    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"]
Jul 30 03:07:39.997335 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:39.997316    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"
Jul 30 03:07:40.000116 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:40.000091    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-cb71ac-serving-cert\""
Jul 30 03:07:40.008437 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:40.008409    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"]
Jul 30 03:07:40.072383 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:40.072336    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78ae001c-3d0d-49e1-a6e1-6497701da69e-maas-api-tls\") pod \"maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz\" (UID: \"78ae001c-3d0d-49e1-a6e1-6497701da69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"
Jul 30 03:07:40.072383 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:40.072384    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw6z4\" (UniqueName: \"kubernetes.io/projected/78ae001c-3d0d-49e1-a6e1-6497701da69e-kube-api-access-nw6z4\") pod \"maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz\" (UID: \"78ae001c-3d0d-49e1-a6e1-6497701da69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"
Jul 30 03:07:40.072650 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:40.072451    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ae001c-3d0d-49e1-a6e1-6497701da69e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz\" (UID: \"78ae001c-3d0d-49e1-a6e1-6497701da69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"
Jul 30 03:07:40.174077 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:40.173983    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ae001c-3d0d-49e1-a6e1-6497701da69e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz\" (UID: \"78ae001c-3d0d-49e1-a6e1-6497701da69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"
Jul 30 03:07:40.174305 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:40.174079    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78ae001c-3d0d-49e1-a6e1-6497701da69e-maas-api-tls\") pod \"maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz\" (UID: \"78ae001c-3d0d-49e1-a6e1-6497701da69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"
Jul 30 03:07:40.174305 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:40.174120    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw6z4\" (UniqueName: \"kubernetes.io/projected/78ae001c-3d0d-49e1-a6e1-6497701da69e-kube-api-access-nw6z4\") pod \"maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz\" (UID: \"78ae001c-3d0d-49e1-a6e1-6497701da69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"
Jul 30 03:07:40.174305 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:07:40.174262    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-serving-cert: secret "maas-api-e2e-inf-b-cb71ac-serving-cert" not found
Jul 30 03:07:40.174481 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:07:40.174332    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78ae001c-3d0d-49e1-a6e1-6497701da69e-maas-api-tls podName:78ae001c-3d0d-49e1-a6e1-6497701da69e nodeName:}" failed. No retries permitted until 2026-07-30 03:07:40.674314191 +0000 UTC m=+2496.321097577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/78ae001c-3d0d-49e1-a6e1-6497701da69e-maas-api-tls") pod "maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz" (UID: "78ae001c-3d0d-49e1-a6e1-6497701da69e") : secret "maas-api-e2e-inf-b-cb71ac-serving-cert" not found
Jul 30 03:07:40.174669 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:40.174649    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ae001c-3d0d-49e1-a6e1-6497701da69e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz\" (UID: \"78ae001c-3d0d-49e1-a6e1-6497701da69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"
Jul 30 03:07:40.183497 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:40.183479    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw6z4\" (UniqueName: \"kubernetes.io/projected/78ae001c-3d0d-49e1-a6e1-6497701da69e-kube-api-access-nw6z4\") pod \"maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz\" (UID: \"78ae001c-3d0d-49e1-a6e1-6497701da69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"
Jul 30 03:07:40.679995 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:40.679938    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78ae001c-3d0d-49e1-a6e1-6497701da69e-maas-api-tls\") pod \"maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz\" (UID: \"78ae001c-3d0d-49e1-a6e1-6497701da69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"
Jul 30 03:07:40.682831 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:40.682800    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78ae001c-3d0d-49e1-a6e1-6497701da69e-maas-api-tls\") pod \"maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz\" (UID: \"78ae001c-3d0d-49e1-a6e1-6497701da69e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"
Jul 30 03:07:40.912889 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:40.912851    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"
Jul 30 03:07:41.063158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:41.063131    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"]
Jul 30 03:07:41.064863 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:07:41.064836    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ae001c_3d0d_49e1_a6e1_6497701da69e.slice/crio-166427a4a1924b88edba4b7a741ff2ed0d78970392206c7a8b4f656435248901 WatchSource:0}: Error finding container 166427a4a1924b88edba4b7a741ff2ed0d78970392206c7a8b4f656435248901: Status 404 returned error can't find the container with id 166427a4a1924b88edba4b7a741ff2ed0d78970392206c7a8b4f656435248901
Jul 30 03:07:41.922097 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:41.922061    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz" event={"ID":"78ae001c-3d0d-49e1-a6e1-6497701da69e","Type":"ContainerStarted","Data":"8a8700d39265d550c884f9a35bd6221afdc7891a6e5bbfdf7d1443b5359cceb9"}
Jul 30 03:07:41.922532 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:41.922124    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"
Jul 30 03:07:41.922532 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:41.922142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz" event={"ID":"78ae001c-3d0d-49e1-a6e1-6497701da69e","Type":"ContainerStarted","Data":"166427a4a1924b88edba4b7a741ff2ed0d78970392206c7a8b4f656435248901"}
Jul 30 03:07:41.940124 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:41.940067    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz" podStartSLOduration=2.46893658 podStartE2EDuration="2.940047352s" totalImagesPullingTime="471.110772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:07:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:07:41.938716715 +0000 UTC m=+2497.585500130" watchObservedRunningTime="2026-07-30 03:07:41.940047352 +0000 UTC m=+2497.586830756"
Jul 30 03:07:44.878425 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:44.878395    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4"
Jul 30 03:07:44.879367 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:44.879338    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t"
Jul 30 03:07:47.892364 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:47.892320    2565 patch_prober.go:28] interesting pod/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 30 03:07:47.894547 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:47.892399    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr" podUID="c83eff9b-ae72-4f2c-8c25-bb333b3713ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 30 03:07:47.934237 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:47.934213    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"
Jul 30 03:07:47.959221 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:47.959152    2565 generic.go:356] "Generic (PLEG): container finished" podID="c83eff9b-ae72-4f2c-8c25-bb333b3713ee" containerID="b619ab6572cf26c6d908d76554898560f06d37e15c27443ab12a125a82ac192e" exitCode=0
Jul 30 03:07:47.959221 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:47.959209    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr" event={"ID":"c83eff9b-ae72-4f2c-8c25-bb333b3713ee","Type":"ContainerDied","Data":"b619ab6572cf26c6d908d76554898560f06d37e15c27443ab12a125a82ac192e"}
Jul 30 03:07:47.959679 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:47.959661    2565 scope.go:117] "RemoveContainer" containerID="b619ab6572cf26c6d908d76554898560f06d37e15c27443ab12a125a82ac192e"
Jul 30 03:07:47.960973 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:47.960940    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:07:47.961049 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:47.961027    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:07:47.961099 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:47.961082    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:07:48.967582 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:48.967546    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr" event={"ID":"c83eff9b-ae72-4f2c-8c25-bb333b3713ee","Type":"ContainerStarted","Data":"40f3ea9c4c28dc7617746d2da648edad8d12d0678a4985b70f7bfc4d63009b11"}
Jul 30 03:07:49.922842 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:49.922806    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:49.923297 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:49.923209    2565 patch_prober.go:28] interesting pod/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Jul 30 03:07:49.923492 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:49.923470    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr" podUID="c83eff9b-ae72-4f2c-8c25-bb333b3713ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Jul 30 03:07:50.922285 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:50.922249    2565 patch_prober.go:28] interesting pod/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Jul 30 03:07:50.923307 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:50.922879    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr" podUID="c83eff9b-ae72-4f2c-8c25-bb333b3713ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Jul 30 03:07:51.922972 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:51.922925    2565 patch_prober.go:28] interesting pod/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Jul 30 03:07:51.923500 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:51.922997    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr" podUID="c83eff9b-ae72-4f2c-8c25-bb333b3713ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Jul 30 03:07:52.922721 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:52.922662    2565 patch_prober.go:28] interesting pod/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Jul 30 03:07:52.922935 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:52.922743    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr" podUID="c83eff9b-ae72-4f2c-8c25-bb333b3713ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Jul 30 03:07:53.925779 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:53.925744    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:53.926141 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:53.926102    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:53.926949 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:53.926924    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:07:54.944384 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:54.944336    2565 patch_prober.go:28] interesting pod/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 30 03:07:54.944845 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:54.944408    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84" podUID="ab96c315-6328-41ed-8ccd-eecfd828f6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 30 03:07:57.033946 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:57.033856    2565 generic.go:356] "Generic (PLEG): container finished" podID="ab96c315-6328-41ed-8ccd-eecfd828f6d9" containerID="b725ee2bb39f83c5dd257f5c8b75b3314f380d5ac9354e4f5de68264d6662591" exitCode=0
Jul 30 03:07:57.033946 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:57.033927    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84" event={"ID":"ab96c315-6328-41ed-8ccd-eecfd828f6d9","Type":"ContainerDied","Data":"b725ee2bb39f83c5dd257f5c8b75b3314f380d5ac9354e4f5de68264d6662591"}
Jul 30 03:07:57.034439 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:57.034392    2565 scope.go:117] "RemoveContainer" containerID="b725ee2bb39f83c5dd257f5c8b75b3314f380d5ac9354e4f5de68264d6662591"
Jul 30 03:07:57.035508 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:57.035463    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:07:57.035612 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:57.035552    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:07:57.035612 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:57.035586    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:07:58.043909 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:58.043872    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84" event={"ID":"ab96c315-6328-41ed-8ccd-eecfd828f6d9","Type":"ContainerStarted","Data":"2ec16e0c88631faa60e0a2535f4c5e612659362c6e727e713f753939d0979acb"}
Jul 30 03:07:58.251747 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:58.251713    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:07:58.251931 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:58.251895    2565 patch_prober.go:28] interesting pod/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Jul 30 03:07:58.251975 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:58.251944    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84" podUID="ab96c315-6328-41ed-8ccd-eecfd828f6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Jul 30 03:07:59.252150 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:59.252116    2565 patch_prober.go:28] interesting pod/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Jul 30 03:07:59.252546 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:07:59.252196    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84" podUID="ab96c315-6328-41ed-8ccd-eecfd828f6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Jul 30 03:08:00.252133 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:08:00.252096    2565 patch_prober.go:28] interesting pod/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Jul 30 03:08:00.252385 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:08:00.252151    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84" podUID="ab96c315-6328-41ed-8ccd-eecfd828f6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Jul 30 03:08:01.251763 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:08:01.251722    2565 patch_prober.go:28] interesting pod/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Jul 30 03:08:01.251947 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:08:01.251825    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84" podUID="ab96c315-6328-41ed-8ccd-eecfd828f6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Jul 30 03:08:02.255485 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:08:02.255453    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:08:02.255971 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:08:02.255873    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:08:02.256574 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:08:02.256553    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:09:05.210626 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:05.210590    2565 scope.go:117] "RemoveContainer" containerID="1a420fa0711c315ceaf26c12776abe7aa2f38174e376efb2e5ec3292e281ef3f"
Jul 30 03:09:15.649939 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:15.649897    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"]
Jul 30 03:09:15.650438 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:15.650314    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw" podUID="56cb47e7-ee70-4010-b03e-afb498527c2c" containerName="maas-api" containerID="cri-o://e9dc7080db493d24f0966ab15699010b2e25b56f3f5042956d7c6af748ceaca1" gracePeriod=30
Jul 30 03:09:15.711493 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:15.711457    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t"]
Jul 30 03:09:15.711754 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:15.711708    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t" podUID="72d9ae38-8e67-43a0-924d-2dd017b7fc58" containerName="payload-processing" containerID="cri-o://7649e46431a5b0ff38ec40de3c571253e9960b66fdaeb0f031dbbf7bc6000022" gracePeriod=30
Jul 30 03:09:15.719134 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:15.719107    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4"]
Jul 30 03:09:15.719398 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:15.719368    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4" podUID="ba1c1018-4261-45fc-b5a4-3c4eabb35768" containerName="payload-pre-processing" containerID="cri-o://4f6a17d42156eddff13a3e59ac4b04facf5a2ea61f6a87c00f8cebed73ed538d" gracePeriod=30
Jul 30 03:09:15.987687 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:15.987625    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"
Jul 30 03:09:16.023570 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.023540    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t"
Jul 30 03:09:16.028357 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.028329    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pzl8\" (UniqueName: \"kubernetes.io/projected/56cb47e7-ee70-4010-b03e-afb498527c2c-kube-api-access-6pzl8\") pod \"56cb47e7-ee70-4010-b03e-afb498527c2c\" (UID: \"56cb47e7-ee70-4010-b03e-afb498527c2c\") "
Jul 30 03:09:16.028502 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.028478    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cb47e7-ee70-4010-b03e-afb498527c2c-openshift-service-ca\") pod \"56cb47e7-ee70-4010-b03e-afb498527c2c\" (UID: \"56cb47e7-ee70-4010-b03e-afb498527c2c\") "
Jul 30 03:09:16.028571 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.028519    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56cb47e7-ee70-4010-b03e-afb498527c2c-maas-api-tls\") pod \"56cb47e7-ee70-4010-b03e-afb498527c2c\" (UID: \"56cb47e7-ee70-4010-b03e-afb498527c2c\") "
Jul 30 03:09:16.028910 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.028888    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56cb47e7-ee70-4010-b03e-afb498527c2c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "56cb47e7-ee70-4010-b03e-afb498527c2c" (UID: "56cb47e7-ee70-4010-b03e-afb498527c2c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:09:16.030724 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.030700    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56cb47e7-ee70-4010-b03e-afb498527c2c-kube-api-access-6pzl8" (OuterVolumeSpecName: "kube-api-access-6pzl8") pod "56cb47e7-ee70-4010-b03e-afb498527c2c" (UID: "56cb47e7-ee70-4010-b03e-afb498527c2c"). InnerVolumeSpecName "kube-api-access-6pzl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:09:16.031020 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.031000    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56cb47e7-ee70-4010-b03e-afb498527c2c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "56cb47e7-ee70-4010-b03e-afb498527c2c" (UID: "56cb47e7-ee70-4010-b03e-afb498527c2c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:09:16.037766 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.037745    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4"
Jul 30 03:09:16.130008 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.129980    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqcfq\" (UniqueName: \"kubernetes.io/projected/72d9ae38-8e67-43a0-924d-2dd017b7fc58-kube-api-access-hqcfq\") pod \"72d9ae38-8e67-43a0-924d-2dd017b7fc58\" (UID: \"72d9ae38-8e67-43a0-924d-2dd017b7fc58\") "
Jul 30 03:09:16.130230 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.130025    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba1c1018-4261-45fc-b5a4-3c4eabb35768-plugins-config-volume\") pod \"ba1c1018-4261-45fc-b5a4-3c4eabb35768\" (UID: \"ba1c1018-4261-45fc-b5a4-3c4eabb35768\") "
Jul 30 03:09:16.130230 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.130045    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d9ae38-8e67-43a0-924d-2dd017b7fc58-plugins-config-volume\") pod \"72d9ae38-8e67-43a0-924d-2dd017b7fc58\" (UID: \"72d9ae38-8e67-43a0-924d-2dd017b7fc58\") "
Jul 30 03:09:16.130230 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.130087    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vv7jl\" (UniqueName: \"kubernetes.io/projected/ba1c1018-4261-45fc-b5a4-3c4eabb35768-kube-api-access-vv7jl\") pod \"ba1c1018-4261-45fc-b5a4-3c4eabb35768\" (UID: \"ba1c1018-4261-45fc-b5a4-3c4eabb35768\") "
Jul 30 03:09:16.130431 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.130349    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pzl8\" (UniqueName: \"kubernetes.io/projected/56cb47e7-ee70-4010-b03e-afb498527c2c-kube-api-access-6pzl8\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:16.130431 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.130363    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56cb47e7-ee70-4010-b03e-afb498527c2c-openshift-service-ca\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:16.130431 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.130372    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56cb47e7-ee70-4010-b03e-afb498527c2c-maas-api-tls\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:16.130431 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.130406    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba1c1018-4261-45fc-b5a4-3c4eabb35768-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ba1c1018-4261-45fc-b5a4-3c4eabb35768" (UID: "ba1c1018-4261-45fc-b5a4-3c4eabb35768"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:09:16.130598 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.130486    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72d9ae38-8e67-43a0-924d-2dd017b7fc58-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "72d9ae38-8e67-43a0-924d-2dd017b7fc58" (UID: "72d9ae38-8e67-43a0-924d-2dd017b7fc58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:09:16.132159 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.132141    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d9ae38-8e67-43a0-924d-2dd017b7fc58-kube-api-access-hqcfq" (OuterVolumeSpecName: "kube-api-access-hqcfq") pod "72d9ae38-8e67-43a0-924d-2dd017b7fc58" (UID: "72d9ae38-8e67-43a0-924d-2dd017b7fc58"). InnerVolumeSpecName "kube-api-access-hqcfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:09:16.132314 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.132297    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1c1018-4261-45fc-b5a4-3c4eabb35768-kube-api-access-vv7jl" (OuterVolumeSpecName: "kube-api-access-vv7jl") pod "ba1c1018-4261-45fc-b5a4-3c4eabb35768" (UID: "ba1c1018-4261-45fc-b5a4-3c4eabb35768"). InnerVolumeSpecName "kube-api-access-vv7jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:09:16.231393 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.231309    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv7jl\" (UniqueName: \"kubernetes.io/projected/ba1c1018-4261-45fc-b5a4-3c4eabb35768-kube-api-access-vv7jl\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:16.231393 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.231338    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqcfq\" (UniqueName: \"kubernetes.io/projected/72d9ae38-8e67-43a0-924d-2dd017b7fc58-kube-api-access-hqcfq\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:16.231393 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.231348    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba1c1018-4261-45fc-b5a4-3c4eabb35768-plugins-config-volume\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:16.231393 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.231356    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d9ae38-8e67-43a0-924d-2dd017b7fc58-plugins-config-volume\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:16.502336 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.502233    2565 generic.go:356] "Generic (PLEG): container finished" podID="56cb47e7-ee70-4010-b03e-afb498527c2c" containerID="e9dc7080db493d24f0966ab15699010b2e25b56f3f5042956d7c6af748ceaca1" exitCode=0
Jul 30 03:09:16.502336 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.502284    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"
Jul 30 03:09:16.502578 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.502324    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw" event={"ID":"56cb47e7-ee70-4010-b03e-afb498527c2c","Type":"ContainerDied","Data":"e9dc7080db493d24f0966ab15699010b2e25b56f3f5042956d7c6af748ceaca1"}
Jul 30 03:09:16.502578 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.502389    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw" event={"ID":"56cb47e7-ee70-4010-b03e-afb498527c2c","Type":"ContainerDied","Data":"f28658ddc569cdf0fa891e511799c1407c90de30fb2f6cb3cc6e25091ca88963"}
Jul 30 03:09:16.502578 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.502417    2565 scope.go:117] "RemoveContainer" containerID="e9dc7080db493d24f0966ab15699010b2e25b56f3f5042956d7c6af748ceaca1"
Jul 30 03:09:16.503789 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.503763    2565 generic.go:356] "Generic (PLEG): container finished" podID="72d9ae38-8e67-43a0-924d-2dd017b7fc58" containerID="7649e46431a5b0ff38ec40de3c571253e9960b66fdaeb0f031dbbf7bc6000022" exitCode=0
Jul 30 03:09:16.503919 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.503829    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t"
Jul 30 03:09:16.503919 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.503865    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t" event={"ID":"72d9ae38-8e67-43a0-924d-2dd017b7fc58","Type":"ContainerDied","Data":"7649e46431a5b0ff38ec40de3c571253e9960b66fdaeb0f031dbbf7bc6000022"}
Jul 30 03:09:16.503919 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.503916    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t" event={"ID":"72d9ae38-8e67-43a0-924d-2dd017b7fc58","Type":"ContainerDied","Data":"ad0c209133d7cdbeb36d6ee44e70220a89893ce0747d6c6247554cf8518e9ce4"}
Jul 30 03:09:16.505393 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.505368    2565 generic.go:356] "Generic (PLEG): container finished" podID="ba1c1018-4261-45fc-b5a4-3c4eabb35768" containerID="4f6a17d42156eddff13a3e59ac4b04facf5a2ea61f6a87c00f8cebed73ed538d" exitCode=0
Jul 30 03:09:16.505507 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.505420    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4"
Jul 30 03:09:16.505507 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.505449    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4" event={"ID":"ba1c1018-4261-45fc-b5a4-3c4eabb35768","Type":"ContainerDied","Data":"4f6a17d42156eddff13a3e59ac4b04facf5a2ea61f6a87c00f8cebed73ed538d"}
Jul 30 03:09:16.505649 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.505513    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4" event={"ID":"ba1c1018-4261-45fc-b5a4-3c4eabb35768","Type":"ContainerDied","Data":"564c3a06f2b7c7d43a6931a92c8dd9827f5e70b2e5fc1a88d5d347e9a14cb3a5"}
Jul 30 03:09:16.513968 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.513938    2565 scope.go:117] "RemoveContainer" containerID="e9dc7080db493d24f0966ab15699010b2e25b56f3f5042956d7c6af748ceaca1"
Jul 30 03:09:16.514284 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:09:16.514258    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9dc7080db493d24f0966ab15699010b2e25b56f3f5042956d7c6af748ceaca1\": container with ID starting with e9dc7080db493d24f0966ab15699010b2e25b56f3f5042956d7c6af748ceaca1 not found: ID does not exist" containerID="e9dc7080db493d24f0966ab15699010b2e25b56f3f5042956d7c6af748ceaca1"
Jul 30 03:09:16.514357 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.514296    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9dc7080db493d24f0966ab15699010b2e25b56f3f5042956d7c6af748ceaca1"} err="failed to get container status \"e9dc7080db493d24f0966ab15699010b2e25b56f3f5042956d7c6af748ceaca1\": rpc error: code = NotFound desc = could not find container \"e9dc7080db493d24f0966ab15699010b2e25b56f3f5042956d7c6af748ceaca1\": container with ID starting with e9dc7080db493d24f0966ab15699010b2e25b56f3f5042956d7c6af748ceaca1 not found: ID does not exist"
Jul 30 03:09:16.514357 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.514334    2565 scope.go:117] "RemoveContainer" containerID="7649e46431a5b0ff38ec40de3c571253e9960b66fdaeb0f031dbbf7bc6000022"
Jul 30 03:09:16.525420 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.525405    2565 scope.go:117] "RemoveContainer" containerID="7649e46431a5b0ff38ec40de3c571253e9960b66fdaeb0f031dbbf7bc6000022"
Jul 30 03:09:16.525666 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:09:16.525648    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7649e46431a5b0ff38ec40de3c571253e9960b66fdaeb0f031dbbf7bc6000022\": container with ID starting with 7649e46431a5b0ff38ec40de3c571253e9960b66fdaeb0f031dbbf7bc6000022 not found: ID does not exist" containerID="7649e46431a5b0ff38ec40de3c571253e9960b66fdaeb0f031dbbf7bc6000022"
Jul 30 03:09:16.525717 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.525672    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7649e46431a5b0ff38ec40de3c571253e9960b66fdaeb0f031dbbf7bc6000022"} err="failed to get container status \"7649e46431a5b0ff38ec40de3c571253e9960b66fdaeb0f031dbbf7bc6000022\": rpc error: code = NotFound desc = could not find container \"7649e46431a5b0ff38ec40de3c571253e9960b66fdaeb0f031dbbf7bc6000022\": container with ID starting with 7649e46431a5b0ff38ec40de3c571253e9960b66fdaeb0f031dbbf7bc6000022 not found: ID does not exist"
Jul 30 03:09:16.525717 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.525686    2565 scope.go:117] "RemoveContainer" containerID="4f6a17d42156eddff13a3e59ac4b04facf5a2ea61f6a87c00f8cebed73ed538d"
Jul 30 03:09:16.535206 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.535187    2565 scope.go:117] "RemoveContainer" containerID="4f6a17d42156eddff13a3e59ac4b04facf5a2ea61f6a87c00f8cebed73ed538d"
Jul 30 03:09:16.535463 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:09:16.535443    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f6a17d42156eddff13a3e59ac4b04facf5a2ea61f6a87c00f8cebed73ed538d\": container with ID starting with 4f6a17d42156eddff13a3e59ac4b04facf5a2ea61f6a87c00f8cebed73ed538d not found: ID does not exist" containerID="4f6a17d42156eddff13a3e59ac4b04facf5a2ea61f6a87c00f8cebed73ed538d"
Jul 30 03:09:16.535539 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.535474    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f6a17d42156eddff13a3e59ac4b04facf5a2ea61f6a87c00f8cebed73ed538d"} err="failed to get container status \"4f6a17d42156eddff13a3e59ac4b04facf5a2ea61f6a87c00f8cebed73ed538d\": rpc error: code = NotFound desc = could not find container \"4f6a17d42156eddff13a3e59ac4b04facf5a2ea61f6a87c00f8cebed73ed538d\": container with ID starting with 4f6a17d42156eddff13a3e59ac4b04facf5a2ea61f6a87c00f8cebed73ed538d not found: ID does not exist"
Jul 30 03:09:16.535539 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.535501    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t"]
Jul 30 03:09:16.538044 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.538025    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3b8039-6b87d47bb8-68b2t"]
Jul 30 03:09:16.552215 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.552182    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"]
Jul 30 03:09:16.555023 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.555001    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3b8039-75dbfd56d4-7ntbw"]
Jul 30 03:09:16.565638 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.565611    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4"]
Jul 30 03:09:16.569854 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.569830    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3b8039-7596f59bbd-tz9l4"]
Jul 30 03:09:16.802042 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.802008    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56cb47e7-ee70-4010-b03e-afb498527c2c" path="/var/lib/kubelet/pods/56cb47e7-ee70-4010-b03e-afb498527c2c/volumes"
Jul 30 03:09:16.802440 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.802381    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72d9ae38-8e67-43a0-924d-2dd017b7fc58" path="/var/lib/kubelet/pods/72d9ae38-8e67-43a0-924d-2dd017b7fc58/volumes"
Jul 30 03:09:16.802687 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:16.802675    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba1c1018-4261-45fc-b5a4-3c4eabb35768" path="/var/lib/kubelet/pods/ba1c1018-4261-45fc-b5a4-3c4eabb35768/volumes"
Jul 30 03:09:26.607791 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:26.607713    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"]
Jul 30 03:09:26.608127 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:26.607960    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr" podUID="c83eff9b-ae72-4f2c-8c25-bb333b3713ee" containerName="istio-proxy" containerID="cri-o://40f3ea9c4c28dc7617746d2da648edad8d12d0678a4985b70f7bfc4d63009b11" gracePeriod=30
Jul 30 03:09:31.865612 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.865588    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:09:31.983395 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.983314    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istiod-ca-cert\") pod \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") "
Jul 30 03:09:31.983395 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.983359    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-credential-socket\") pod \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") "
Jul 30 03:09:31.983395 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.983383    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-workload-certs\") pod \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") "
Jul 30 03:09:31.983673 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.983418    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-data\") pod \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") "
Jul 30 03:09:31.983673 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.983476    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-envoy\") pod \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") "
Jul 30 03:09:31.983673 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.983515    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-token\") pod \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") "
Jul 30 03:09:31.983673 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.983570    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sc5pm\" (UniqueName: \"kubernetes.io/projected/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-kube-api-access-sc5pm\") pod \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") "
Jul 30 03:09:31.983673 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.983603    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-podinfo\") pod \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") "
Jul 30 03:09:31.983673 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.983628    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c83eff9b-ae72-4f2c-8c25-bb333b3713ee" (UID: "c83eff9b-ae72-4f2c-8c25-bb333b3713ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:09:31.983673 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.983646    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-workload-socket\") pod \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\" (UID: \"c83eff9b-ae72-4f2c-8c25-bb333b3713ee\") "
Jul 30 03:09:31.984020 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.983687    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c83eff9b-ae72-4f2c-8c25-bb333b3713ee" (UID: "c83eff9b-ae72-4f2c-8c25-bb333b3713ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:09:31.984020 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.983805    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c83eff9b-ae72-4f2c-8c25-bb333b3713ee" (UID: "c83eff9b-ae72-4f2c-8c25-bb333b3713ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:09:31.984144 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.984022    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c83eff9b-ae72-4f2c-8c25-bb333b3713ee" (UID: "c83eff9b-ae72-4f2c-8c25-bb333b3713ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:09:31.984144 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.984105    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-workload-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:31.984144 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.984124    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istiod-ca-cert\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:31.984144 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.984137    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-credential-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:31.984425 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.984150    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-workload-certs\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:31.984985 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.984958    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-data" (OuterVolumeSpecName: "istio-data") pod "c83eff9b-ae72-4f2c-8c25-bb333b3713ee" (UID: "c83eff9b-ae72-4f2c-8c25-bb333b3713ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:09:31.986024 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.985995    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c83eff9b-ae72-4f2c-8c25-bb333b3713ee" (UID: "c83eff9b-ae72-4f2c-8c25-bb333b3713ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:09:31.986099 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.986031    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-kube-api-access-sc5pm" (OuterVolumeSpecName: "kube-api-access-sc5pm") pod "c83eff9b-ae72-4f2c-8c25-bb333b3713ee" (UID: "c83eff9b-ae72-4f2c-8c25-bb333b3713ee"). InnerVolumeSpecName "kube-api-access-sc5pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:09:31.986139 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.986105    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-token" (OuterVolumeSpecName: "istio-token") pod "c83eff9b-ae72-4f2c-8c25-bb333b3713ee" (UID: "c83eff9b-ae72-4f2c-8c25-bb333b3713ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:09:31.986402 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:31.986387    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c83eff9b-ae72-4f2c-8c25-bb333b3713ee" (UID: "c83eff9b-ae72-4f2c-8c25-bb333b3713ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 30 03:09:32.084940 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:32.084904    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-envoy\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:32.084940 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:32.084933    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-token\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:32.085153 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:32.084953    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc5pm\" (UniqueName: \"kubernetes.io/projected/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-kube-api-access-sc5pm\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:32.085153 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:32.084963    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-podinfo\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:32.085153 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:32.084971    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c83eff9b-ae72-4f2c-8c25-bb333b3713ee-istio-data\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:32.603387 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:32.603352    2565 generic.go:356] "Generic (PLEG): container finished" podID="c83eff9b-ae72-4f2c-8c25-bb333b3713ee" containerID="40f3ea9c4c28dc7617746d2da648edad8d12d0678a4985b70f7bfc4d63009b11" exitCode=0
Jul 30 03:09:32.603569 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:32.603400    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"
Jul 30 03:09:32.603569 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:32.603443    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr" event={"ID":"c83eff9b-ae72-4f2c-8c25-bb333b3713ee","Type":"ContainerDied","Data":"40f3ea9c4c28dc7617746d2da648edad8d12d0678a4985b70f7bfc4d63009b11"}
Jul 30 03:09:32.603569 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:32.603506    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr" event={"ID":"c83eff9b-ae72-4f2c-8c25-bb333b3713ee","Type":"ContainerDied","Data":"6a52a9f5511d278ba12a9dada657f3b3674354ab9bab36ee9b10e2943dc9c19b"}
Jul 30 03:09:32.603569 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:32.603536    2565 scope.go:117] "RemoveContainer" containerID="40f3ea9c4c28dc7617746d2da648edad8d12d0678a4985b70f7bfc4d63009b11"
Jul 30 03:09:32.613550 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:32.613529    2565 scope.go:117] "RemoveContainer" containerID="b619ab6572cf26c6d908d76554898560f06d37e15c27443ab12a125a82ac192e"
Jul 30 03:09:32.622145 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:32.622128    2565 scope.go:117] "RemoveContainer" containerID="40f3ea9c4c28dc7617746d2da648edad8d12d0678a4985b70f7bfc4d63009b11"
Jul 30 03:09:32.622395 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:09:32.622379    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f3ea9c4c28dc7617746d2da648edad8d12d0678a4985b70f7bfc4d63009b11\": container with ID starting with 40f3ea9c4c28dc7617746d2da648edad8d12d0678a4985b70f7bfc4d63009b11 not found: ID does not exist" containerID="40f3ea9c4c28dc7617746d2da648edad8d12d0678a4985b70f7bfc4d63009b11"
Jul 30 03:09:32.622459 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:32.622402    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f3ea9c4c28dc7617746d2da648edad8d12d0678a4985b70f7bfc4d63009b11"} err="failed to get container status \"40f3ea9c4c28dc7617746d2da648edad8d12d0678a4985b70f7bfc4d63009b11\": rpc error: code = NotFound desc = could not find container \"40f3ea9c4c28dc7617746d2da648edad8d12d0678a4985b70f7bfc4d63009b11\": container with ID starting with 40f3ea9c4c28dc7617746d2da648edad8d12d0678a4985b70f7bfc4d63009b11 not found: ID does not exist"
Jul 30 03:09:32.622459 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:32.622418    2565 scope.go:117] "RemoveContainer" containerID="b619ab6572cf26c6d908d76554898560f06d37e15c27443ab12a125a82ac192e"
Jul 30 03:09:32.622688 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:09:32.622669    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b619ab6572cf26c6d908d76554898560f06d37e15c27443ab12a125a82ac192e\": container with ID starting with b619ab6572cf26c6d908d76554898560f06d37e15c27443ab12a125a82ac192e not found: ID does not exist" containerID="b619ab6572cf26c6d908d76554898560f06d37e15c27443ab12a125a82ac192e"
Jul 30 03:09:32.622777 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:32.622691    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b619ab6572cf26c6d908d76554898560f06d37e15c27443ab12a125a82ac192e"} err="failed to get container status \"b619ab6572cf26c6d908d76554898560f06d37e15c27443ab12a125a82ac192e\": rpc error: code = NotFound desc = could not find container \"b619ab6572cf26c6d908d76554898560f06d37e15c27443ab12a125a82ac192e\": container with ID starting with b619ab6572cf26c6d908d76554898560f06d37e15c27443ab12a125a82ac192e not found: ID does not exist"
Jul 30 03:09:32.626593 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:32.626569    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"]
Jul 30 03:09:32.630032 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:32.630009    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3b8039-openshift-default-5c6c4ccc68-9nblr"]
Jul 30 03:09:32.800918 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:32.800881    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c83eff9b-ae72-4f2c-8c25-bb333b3713ee" path="/var/lib/kubelet/pods/c83eff9b-ae72-4f2c-8c25-bb333b3713ee/volumes"
Jul 30 03:09:37.387487 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:37.387441    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"]
Jul 30 03:09:37.388079 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:37.387726    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz" podUID="78ae001c-3d0d-49e1-a6e1-6497701da69e" containerName="maas-api" containerID="cri-o://8a8700d39265d550c884f9a35bd6221afdc7891a6e5bbfdf7d1443b5359cceb9" gracePeriod=30
Jul 30 03:09:37.849648 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:37.849621    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"
Jul 30 03:09:37.941919 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:37.941834    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw6z4\" (UniqueName: \"kubernetes.io/projected/78ae001c-3d0d-49e1-a6e1-6497701da69e-kube-api-access-nw6z4\") pod \"78ae001c-3d0d-49e1-a6e1-6497701da69e\" (UID: \"78ae001c-3d0d-49e1-a6e1-6497701da69e\") "
Jul 30 03:09:37.942075 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:37.941929    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ae001c-3d0d-49e1-a6e1-6497701da69e-openshift-service-ca\") pod \"78ae001c-3d0d-49e1-a6e1-6497701da69e\" (UID: \"78ae001c-3d0d-49e1-a6e1-6497701da69e\") "
Jul 30 03:09:37.942075 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:37.941952    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78ae001c-3d0d-49e1-a6e1-6497701da69e-maas-api-tls\") pod \"78ae001c-3d0d-49e1-a6e1-6497701da69e\" (UID: \"78ae001c-3d0d-49e1-a6e1-6497701da69e\") "
Jul 30 03:09:37.942454 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:37.942276    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ae001c-3d0d-49e1-a6e1-6497701da69e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "78ae001c-3d0d-49e1-a6e1-6497701da69e" (UID: "78ae001c-3d0d-49e1-a6e1-6497701da69e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:09:37.944246 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:37.944225    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78ae001c-3d0d-49e1-a6e1-6497701da69e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "78ae001c-3d0d-49e1-a6e1-6497701da69e" (UID: "78ae001c-3d0d-49e1-a6e1-6497701da69e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:09:37.944327 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:37.944297    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ae001c-3d0d-49e1-a6e1-6497701da69e-kube-api-access-nw6z4" (OuterVolumeSpecName: "kube-api-access-nw6z4") pod "78ae001c-3d0d-49e1-a6e1-6497701da69e" (UID: "78ae001c-3d0d-49e1-a6e1-6497701da69e"). InnerVolumeSpecName "kube-api-access-nw6z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:09:38.043650 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:38.043613    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78ae001c-3d0d-49e1-a6e1-6497701da69e-openshift-service-ca\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:38.043650 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:38.043643    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78ae001c-3d0d-49e1-a6e1-6497701da69e-maas-api-tls\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:38.043650 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:38.043653    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw6z4\" (UniqueName: \"kubernetes.io/projected/78ae001c-3d0d-49e1-a6e1-6497701da69e-kube-api-access-nw6z4\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:38.647131 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:38.647094    2565 generic.go:356] "Generic (PLEG): container finished" podID="78ae001c-3d0d-49e1-a6e1-6497701da69e" containerID="8a8700d39265d550c884f9a35bd6221afdc7891a6e5bbfdf7d1443b5359cceb9" exitCode=0
Jul 30 03:09:38.647558 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:38.647147    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"
Jul 30 03:09:38.647558 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:38.647206    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz" event={"ID":"78ae001c-3d0d-49e1-a6e1-6497701da69e","Type":"ContainerDied","Data":"8a8700d39265d550c884f9a35bd6221afdc7891a6e5bbfdf7d1443b5359cceb9"}
Jul 30 03:09:38.647558 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:38.647261    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz" event={"ID":"78ae001c-3d0d-49e1-a6e1-6497701da69e","Type":"ContainerDied","Data":"166427a4a1924b88edba4b7a741ff2ed0d78970392206c7a8b4f656435248901"}
Jul 30 03:09:38.647558 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:38.647284    2565 scope.go:117] "RemoveContainer" containerID="8a8700d39265d550c884f9a35bd6221afdc7891a6e5bbfdf7d1443b5359cceb9"
Jul 30 03:09:38.658894 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:38.658873    2565 scope.go:117] "RemoveContainer" containerID="8a8700d39265d550c884f9a35bd6221afdc7891a6e5bbfdf7d1443b5359cceb9"
Jul 30 03:09:38.659185 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:09:38.659143    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a8700d39265d550c884f9a35bd6221afdc7891a6e5bbfdf7d1443b5359cceb9\": container with ID starting with 8a8700d39265d550c884f9a35bd6221afdc7891a6e5bbfdf7d1443b5359cceb9 not found: ID does not exist" containerID="8a8700d39265d550c884f9a35bd6221afdc7891a6e5bbfdf7d1443b5359cceb9"
Jul 30 03:09:38.659261 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:38.659195    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a8700d39265d550c884f9a35bd6221afdc7891a6e5bbfdf7d1443b5359cceb9"} err="failed to get container status \"8a8700d39265d550c884f9a35bd6221afdc7891a6e5bbfdf7d1443b5359cceb9\": rpc error: code = NotFound desc = could not find container \"8a8700d39265d550c884f9a35bd6221afdc7891a6e5bbfdf7d1443b5359cceb9\": container with ID starting with 8a8700d39265d550c884f9a35bd6221afdc7891a6e5bbfdf7d1443b5359cceb9 not found: ID does not exist"
Jul 30 03:09:38.675192 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:38.675147    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"]
Jul 30 03:09:38.682222 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:38.682197    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cb71ac-574cc4d9c5-vkwnz"]
Jul 30 03:09:38.802641 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:38.802603    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78ae001c-3d0d-49e1-a6e1-6497701da69e" path="/var/lib/kubelet/pods/78ae001c-3d0d-49e1-a6e1-6497701da69e/volumes"
Jul 30 03:09:48.956977 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:48.956940    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"]
Jul 30 03:09:48.957400 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:48.957144    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84" podUID="ab96c315-6328-41ed-8ccd-eecfd828f6d9" containerName="istio-proxy" containerID="cri-o://2ec16e0c88631faa60e0a2535f4c5e612659362c6e727e713f753939d0979acb" gracePeriod=30
Jul 30 03:09:49.862812 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:49.862770    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"]
Jul 30 03:09:49.863192 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:49.863178    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56cb47e7-ee70-4010-b03e-afb498527c2c" containerName="maas-api"
Jul 30 03:09:49.863243 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:49.863195    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="56cb47e7-ee70-4010-b03e-afb498527c2c" containerName="maas-api"
Jul 30 03:09:49.863243 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:49.863206    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78ae001c-3d0d-49e1-a6e1-6497701da69e" containerName="maas-api"
Jul 30 03:09:49.863243 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:49.863212    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="78ae001c-3d0d-49e1-a6e1-6497701da69e" containerName="maas-api"
Jul 30 03:09:49.863243 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:49.863219    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c83eff9b-ae72-4f2c-8c25-bb333b3713ee" containerName="istio-proxy"
Jul 30 03:09:49.863243 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:49.863225    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c83eff9b-ae72-4f2c-8c25-bb333b3713ee" containerName="istio-proxy"
Jul 30 03:09:49.863243 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:49.863241    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba1c1018-4261-45fc-b5a4-3c4eabb35768" containerName="payload-pre-processing"
Jul 30 03:09:49.863429 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:49.863248    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba1c1018-4261-45fc-b5a4-3c4eabb35768" containerName="payload-pre-processing"
Jul 30 03:09:49.863429 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:49.863255    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72d9ae38-8e67-43a0-924d-2dd017b7fc58" containerName="payload-processing"
Jul 30 03:09:49.863429 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:49.863261    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="72d9ae38-8e67-43a0-924d-2dd017b7fc58" containerName="payload-processing"
Jul 30 03:09:49.863429 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:49.863322    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="78ae001c-3d0d-49e1-a6e1-6497701da69e" containerName="maas-api"
Jul 30 03:09:49.863429 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:49.863331    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="56cb47e7-ee70-4010-b03e-afb498527c2c" containerName="maas-api"
Jul 30 03:09:49.863429 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:49.863336    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c83eff9b-ae72-4f2c-8c25-bb333b3713ee" containerName="istio-proxy"
Jul 30 03:09:49.863429 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:49.863343    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="72d9ae38-8e67-43a0-924d-2dd017b7fc58" containerName="payload-processing"
Jul 30 03:09:49.863429 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:49.863349    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c83eff9b-ae72-4f2c-8c25-bb333b3713ee" containerName="istio-proxy"
Jul 30 03:09:49.863429 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:49.863355    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba1c1018-4261-45fc-b5a4-3c4eabb35768" containerName="payload-pre-processing"
Jul 30 03:09:49.866531 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:49.866513    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:49.873883 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:49.873861    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-970a97-openshift-default-dockercfg-zbk5m\""
Jul 30 03:09:49.903615 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:49.903590    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"]
Jul 30 03:09:50.055184 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.055132    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istiod-ca-cert\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.055662 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.055229    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-envoy\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.055662 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.055256    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-token\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.055662 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.055331    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l4r5\" (UniqueName: \"kubernetes.io/projected/8e09af48-cdfd-492c-83f6-0f18b7e7469f-kube-api-access-6l4r5\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.055662 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.055392    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-workload-certs\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.055662 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.055428    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-data\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.055662 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.055479    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-workload-socket\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.055662 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.055503    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-credential-socket\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.055662 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.055533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-podinfo\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.156348 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.156254    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istiod-ca-cert\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.156348 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.156314    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-envoy\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.156348 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.156337    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-token\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.156660 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.156376    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l4r5\" (UniqueName: \"kubernetes.io/projected/8e09af48-cdfd-492c-83f6-0f18b7e7469f-kube-api-access-6l4r5\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.156660 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.156413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-workload-certs\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.156660 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.156539    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-data\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.156660 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.156597    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-workload-socket\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.156660 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.156624    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-credential-socket\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.156660 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.156654    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-podinfo\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.156984 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.156794    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-workload-certs\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.156984 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.156918    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-data\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.156984 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.156965    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-workload-socket\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.157103 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.157076    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istiod-ca-cert\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.157153 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.157135    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-credential-socket\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.158850 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.158830    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-envoy\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.159152 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.159135    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-podinfo\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.168128 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.168097    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l4r5\" (UniqueName: \"kubernetes.io/projected/8e09af48-cdfd-492c-83f6-0f18b7e7469f-kube-api-access-6l4r5\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.168270 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.168227    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-token\") pod \"e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") " pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.177110 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.177065    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:50.327431 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.327390    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"]
Jul 30 03:09:50.328043 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:09:50.328012    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e09af48_cdfd_492c_83f6_0f18b7e7469f.slice/crio-a0f4de8ac507433f396f2993c2945d800a761ebe03cf7f2361c0405dee42cd66 WatchSource:0}: Error finding container a0f4de8ac507433f396f2993c2945d800a761ebe03cf7f2361c0405dee42cd66: Status 404 returned error can't find the container with id a0f4de8ac507433f396f2993c2945d800a761ebe03cf7f2361c0405dee42cd66
Jul 30 03:09:50.330528 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.330496    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:09:50.330607 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.330563    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:09:50.330607 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.330588    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:09:50.721984 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.721944    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2" event={"ID":"8e09af48-cdfd-492c-83f6-0f18b7e7469f","Type":"ContainerStarted","Data":"d590c4bca6f856017cbdd8ae29594d414c3327ca24de26f2fc84de87552a9b86"}
Jul 30 03:09:50.722187 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.722008    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2" event={"ID":"8e09af48-cdfd-492c-83f6-0f18b7e7469f","Type":"ContainerStarted","Data":"a0f4de8ac507433f396f2993c2945d800a761ebe03cf7f2361c0405dee42cd66"}
Jul 30 03:09:50.748664 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:50.748597    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2" podStartSLOduration=1.748581858 podStartE2EDuration="1.748581858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:09:50.746112406 +0000 UTC m=+2626.392895810" watchObservedRunningTime="2026-07-30 03:09:50.748581858 +0000 UTC m=+2626.395365343"
Jul 30 03:09:51.177901 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:51.177858    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:51.183489 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:51.183457    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:51.728256 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:51.728220    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:51.729399 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:51.729376    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:09:52.861336 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:52.861293    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"]
Jul 30 03:09:52.861943 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:52.861921    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c83eff9b-ae72-4f2c-8c25-bb333b3713ee" containerName="istio-proxy"
Jul 30 03:09:52.862007 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:52.861948    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c83eff9b-ae72-4f2c-8c25-bb333b3713ee" containerName="istio-proxy"
Jul 30 03:09:52.865673 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:52.865652    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"
Jul 30 03:09:52.868871 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:52.868851    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-970a97-serving-cert\""
Jul 30 03:09:52.880247 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:52.880215    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"]
Jul 30 03:09:52.979058 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:52.979031    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411c6990-f13c-436a-85b3-e5aa308421d1-maas-api-tls\") pod \"maas-api-e2e-api-a-970a97-c44dc9595-v89b4\" (UID: \"411c6990-f13c-436a-85b3-e5aa308421d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"
Jul 30 03:09:52.979275 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:52.979082    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k48w\" (UniqueName: \"kubernetes.io/projected/411c6990-f13c-436a-85b3-e5aa308421d1-kube-api-access-4k48w\") pod \"maas-api-e2e-api-a-970a97-c44dc9595-v89b4\" (UID: \"411c6990-f13c-436a-85b3-e5aa308421d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"
Jul 30 03:09:52.979275 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:52.979128    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411c6990-f13c-436a-85b3-e5aa308421d1-openshift-service-ca\") pod \"maas-api-e2e-api-a-970a97-c44dc9595-v89b4\" (UID: \"411c6990-f13c-436a-85b3-e5aa308421d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"
Jul 30 03:09:53.080129 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:53.080088    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411c6990-f13c-436a-85b3-e5aa308421d1-maas-api-tls\") pod \"maas-api-e2e-api-a-970a97-c44dc9595-v89b4\" (UID: \"411c6990-f13c-436a-85b3-e5aa308421d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"
Jul 30 03:09:53.080340 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:53.080137    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k48w\" (UniqueName: \"kubernetes.io/projected/411c6990-f13c-436a-85b3-e5aa308421d1-kube-api-access-4k48w\") pod \"maas-api-e2e-api-a-970a97-c44dc9595-v89b4\" (UID: \"411c6990-f13c-436a-85b3-e5aa308421d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"
Jul 30 03:09:53.080340 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:53.080182    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411c6990-f13c-436a-85b3-e5aa308421d1-openshift-service-ca\") pod \"maas-api-e2e-api-a-970a97-c44dc9595-v89b4\" (UID: \"411c6990-f13c-436a-85b3-e5aa308421d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"
Jul 30 03:09:53.080835 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:53.080812    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411c6990-f13c-436a-85b3-e5aa308421d1-openshift-service-ca\") pod \"maas-api-e2e-api-a-970a97-c44dc9595-v89b4\" (UID: \"411c6990-f13c-436a-85b3-e5aa308421d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"
Jul 30 03:09:53.082853 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:53.082833    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411c6990-f13c-436a-85b3-e5aa308421d1-maas-api-tls\") pod \"maas-api-e2e-api-a-970a97-c44dc9595-v89b4\" (UID: \"411c6990-f13c-436a-85b3-e5aa308421d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"
Jul 30 03:09:53.094720 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:53.094689    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k48w\" (UniqueName: \"kubernetes.io/projected/411c6990-f13c-436a-85b3-e5aa308421d1-kube-api-access-4k48w\") pod \"maas-api-e2e-api-a-970a97-c44dc9595-v89b4\" (UID: \"411c6990-f13c-436a-85b3-e5aa308421d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"
Jul 30 03:09:53.178576 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:53.178471    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"
Jul 30 03:09:53.330517 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:53.325549    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"]
Jul 30 03:09:53.331571 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:53.331056    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 03:09:53.713725 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:53.713689    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q"]
Jul 30 03:09:53.718395 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:53.718379    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q"
Jul 30 03:09:53.723129 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:53.723110    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-970a97\""
Jul 30 03:09:53.741542 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:53.741486    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4" event={"ID":"411c6990-f13c-436a-85b3-e5aa308421d1","Type":"ContainerStarted","Data":"6ae88457178d86b913512d98b1ddc685f33e10df51a7f69e61321d2c325efdfe"}
Jul 30 03:09:53.742577 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:53.742553    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q"]
Jul 30 03:09:53.786601 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:53.786561    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53b57a2c-c367-41b8-b356-da81ec88306f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q\" (UID: \"53b57a2c-c367-41b8-b356-da81ec88306f\") " pod="openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q"
Jul 30 03:09:53.786782 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:53.786724    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbwz8\" (UniqueName: \"kubernetes.io/projected/53b57a2c-c367-41b8-b356-da81ec88306f-kube-api-access-rbwz8\") pod \"payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q\" (UID: \"53b57a2c-c367-41b8-b356-da81ec88306f\") " pod="openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q"
Jul 30 03:09:53.887319 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:53.887287    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53b57a2c-c367-41b8-b356-da81ec88306f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q\" (UID: \"53b57a2c-c367-41b8-b356-da81ec88306f\") " pod="openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q"
Jul 30 03:09:53.887648 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:53.887393    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbwz8\" (UniqueName: \"kubernetes.io/projected/53b57a2c-c367-41b8-b356-da81ec88306f-kube-api-access-rbwz8\") pod \"payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q\" (UID: \"53b57a2c-c367-41b8-b356-da81ec88306f\") " pod="openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q"
Jul 30 03:09:53.888244 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:53.888220    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53b57a2c-c367-41b8-b356-da81ec88306f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q\" (UID: \"53b57a2c-c367-41b8-b356-da81ec88306f\") " pod="openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q"
Jul 30 03:09:53.897813 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:53.897789    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbwz8\" (UniqueName: \"kubernetes.io/projected/53b57a2c-c367-41b8-b356-da81ec88306f-kube-api-access-rbwz8\") pod \"payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q\" (UID: \"53b57a2c-c367-41b8-b356-da81ec88306f\") " pod="openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q"
Jul 30 03:09:54.028871 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.028840    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q"
Jul 30 03:09:54.253393 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.253362    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:09:54.292556 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.292529    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-data\") pod \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") "
Jul 30 03:09:54.292758 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.292565    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tr47q\" (UniqueName: \"kubernetes.io/projected/ab96c315-6328-41ed-8ccd-eecfd828f6d9-kube-api-access-tr47q\") pod \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") "
Jul 30 03:09:54.292758 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.292583    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-token\") pod \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") "
Jul 30 03:09:54.292758 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.292624    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-envoy\") pod \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") "
Jul 30 03:09:54.292758 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.292671    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-credential-socket\") pod \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") "
Jul 30 03:09:54.292758 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.292701    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-podinfo\") pod \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") "
Jul 30 03:09:54.292758 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.292728    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-workload-certs\") pod \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") "
Jul 30 03:09:54.293087 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.292785    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istiod-ca-cert\") pod \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") "
Jul 30 03:09:54.293087 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.292809    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-workload-socket\") pod \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\" (UID: \"ab96c315-6328-41ed-8ccd-eecfd828f6d9\") "
Jul 30 03:09:54.293439 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.293411    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ab96c315-6328-41ed-8ccd-eecfd828f6d9" (UID: "ab96c315-6328-41ed-8ccd-eecfd828f6d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:09:54.293706 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.293594    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ab96c315-6328-41ed-8ccd-eecfd828f6d9" (UID: "ab96c315-6328-41ed-8ccd-eecfd828f6d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:09:54.293706 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.293608    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ab96c315-6328-41ed-8ccd-eecfd828f6d9" (UID: "ab96c315-6328-41ed-8ccd-eecfd828f6d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:09:54.293706 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.293637    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ab96c315-6328-41ed-8ccd-eecfd828f6d9" (UID: "ab96c315-6328-41ed-8ccd-eecfd828f6d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:09:54.293998 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.293948    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-data" (OuterVolumeSpecName: "istio-data") pod "ab96c315-6328-41ed-8ccd-eecfd828f6d9" (UID: "ab96c315-6328-41ed-8ccd-eecfd828f6d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:09:54.296079 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.296045    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-token" (OuterVolumeSpecName: "istio-token") pod "ab96c315-6328-41ed-8ccd-eecfd828f6d9" (UID: "ab96c315-6328-41ed-8ccd-eecfd828f6d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:09:54.296079 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.296064    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ab96c315-6328-41ed-8ccd-eecfd828f6d9" (UID: "ab96c315-6328-41ed-8ccd-eecfd828f6d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:09:54.296273 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.296157    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab96c315-6328-41ed-8ccd-eecfd828f6d9-kube-api-access-tr47q" (OuterVolumeSpecName: "kube-api-access-tr47q") pod "ab96c315-6328-41ed-8ccd-eecfd828f6d9" (UID: "ab96c315-6328-41ed-8ccd-eecfd828f6d9"). InnerVolumeSpecName "kube-api-access-tr47q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:09:54.296273 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.296208    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ab96c315-6328-41ed-8ccd-eecfd828f6d9" (UID: "ab96c315-6328-41ed-8ccd-eecfd828f6d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 30 03:09:54.391428 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.391396    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q"]
Jul 30 03:09:54.392147 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:09:54.392119    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b57a2c_c367_41b8_b356_da81ec88306f.slice/crio-a11f522007cafbde692838f98794af22e3852eac11e1358da1ea2b6badbf82ca WatchSource:0}: Error finding container a11f522007cafbde692838f98794af22e3852eac11e1358da1ea2b6badbf82ca: Status 404 returned error can't find the container with id a11f522007cafbde692838f98794af22e3852eac11e1358da1ea2b6badbf82ca
Jul 30 03:09:54.393593 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.393567    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-workload-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:54.393703 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.393595    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-data\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:54.393703 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.393612    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr47q\" (UniqueName: \"kubernetes.io/projected/ab96c315-6328-41ed-8ccd-eecfd828f6d9-kube-api-access-tr47q\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:54.393703 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.393627    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-token\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:54.393703 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.393640    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-envoy\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:54.393703 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.393649    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-credential-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:54.393703 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.393657    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istio-podinfo\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:54.393703 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.393665    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab96c315-6328-41ed-8ccd-eecfd828f6d9-workload-certs\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:54.393703 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.393672    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab96c315-6328-41ed-8ccd-eecfd828f6d9-istiod-ca-cert\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:09:54.749293 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.749250    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4" event={"ID":"411c6990-f13c-436a-85b3-e5aa308421d1","Type":"ContainerStarted","Data":"28ad4bacc3eaee101f71cefefc0fe6c03c604b11f4bb1e2eb64d5752dc938afb"}
Jul 30 03:09:54.749512 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.749354    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"
Jul 30 03:09:54.751010 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.750984    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q" event={"ID":"53b57a2c-c367-41b8-b356-da81ec88306f","Type":"ContainerStarted","Data":"aade64ad0332a3e586173ec67ea16152ed23e5d066e8668064b3ae7c7fdfc296"}
Jul 30 03:09:54.751149 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.751031    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q" event={"ID":"53b57a2c-c367-41b8-b356-da81ec88306f","Type":"ContainerStarted","Data":"a11f522007cafbde692838f98794af22e3852eac11e1358da1ea2b6badbf82ca"}
Jul 30 03:09:54.751149 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.751062    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q"
Jul 30 03:09:54.752606 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.752581    2565 generic.go:356] "Generic (PLEG): container finished" podID="ab96c315-6328-41ed-8ccd-eecfd828f6d9" containerID="2ec16e0c88631faa60e0a2535f4c5e612659362c6e727e713f753939d0979acb" exitCode=0
Jul 30 03:09:54.752702 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.752618    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84" event={"ID":"ab96c315-6328-41ed-8ccd-eecfd828f6d9","Type":"ContainerDied","Data":"2ec16e0c88631faa60e0a2535f4c5e612659362c6e727e713f753939d0979acb"}
Jul 30 03:09:54.752702 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.752628    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"
Jul 30 03:09:54.752702 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.752662    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84" event={"ID":"ab96c315-6328-41ed-8ccd-eecfd828f6d9","Type":"ContainerDied","Data":"64d79e464865df552f201d2379b06378e5e2d2e441788ec089c053853a575031"}
Jul 30 03:09:54.752702 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.752692    2565 scope.go:117] "RemoveContainer" containerID="2ec16e0c88631faa60e0a2535f4c5e612659362c6e727e713f753939d0979acb"
Jul 30 03:09:54.763351 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.763323    2565 scope.go:117] "RemoveContainer" containerID="b725ee2bb39f83c5dd257f5c8b75b3314f380d5ac9354e4f5de68264d6662591"
Jul 30 03:09:54.772309 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.772286    2565 scope.go:117] "RemoveContainer" containerID="2ec16e0c88631faa60e0a2535f4c5e612659362c6e727e713f753939d0979acb"
Jul 30 03:09:54.772683 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:09:54.772657    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ec16e0c88631faa60e0a2535f4c5e612659362c6e727e713f753939d0979acb\": container with ID starting with 2ec16e0c88631faa60e0a2535f4c5e612659362c6e727e713f753939d0979acb not found: ID does not exist" containerID="2ec16e0c88631faa60e0a2535f4c5e612659362c6e727e713f753939d0979acb"
Jul 30 03:09:54.772774 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.772697    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ec16e0c88631faa60e0a2535f4c5e612659362c6e727e713f753939d0979acb"} err="failed to get container status \"2ec16e0c88631faa60e0a2535f4c5e612659362c6e727e713f753939d0979acb\": rpc error: code = NotFound desc = could not find container \"2ec16e0c88631faa60e0a2535f4c5e612659362c6e727e713f753939d0979acb\": container with ID starting with 2ec16e0c88631faa60e0a2535f4c5e612659362c6e727e713f753939d0979acb not found: ID does not exist"
Jul 30 03:09:54.772774 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.772718    2565 scope.go:117] "RemoveContainer" containerID="b725ee2bb39f83c5dd257f5c8b75b3314f380d5ac9354e4f5de68264d6662591"
Jul 30 03:09:54.773027 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:09:54.773008    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b725ee2bb39f83c5dd257f5c8b75b3314f380d5ac9354e4f5de68264d6662591\": container with ID starting with b725ee2bb39f83c5dd257f5c8b75b3314f380d5ac9354e4f5de68264d6662591 not found: ID does not exist" containerID="b725ee2bb39f83c5dd257f5c8b75b3314f380d5ac9354e4f5de68264d6662591"
Jul 30 03:09:54.773105 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.773032    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b725ee2bb39f83c5dd257f5c8b75b3314f380d5ac9354e4f5de68264d6662591"} err="failed to get container status \"b725ee2bb39f83c5dd257f5c8b75b3314f380d5ac9354e4f5de68264d6662591\": rpc error: code = NotFound desc = could not find container \"b725ee2bb39f83c5dd257f5c8b75b3314f380d5ac9354e4f5de68264d6662591\": container with ID starting with b725ee2bb39f83c5dd257f5c8b75b3314f380d5ac9354e4f5de68264d6662591 not found: ID does not exist"
Jul 30 03:09:54.779267 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.779210    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4" podStartSLOduration=2.227808643 podStartE2EDuration="2.779189687s" totalImagesPullingTime="551.381044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:09:54.77757458 +0000 UTC m=+2630.424357987" watchObservedRunningTime="2026-07-30 03:09:54.779189687 +0000 UTC m=+2630.425973095"
Jul 30 03:09:54.802338 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.802277    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"]
Jul 30 03:09:54.805131 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.805105    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cb71ac-openshift-default-6d897cf694-9wn84"]
Jul 30 03:09:54.827236 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:54.827011    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q" podStartSLOduration=1.826992144 podStartE2EDuration="1.826992144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:09:54.824889709 +0000 UTC m=+2630.471673115" watchObservedRunningTime="2026-07-30 03:09:54.826992144 +0000 UTC m=+2630.473775550"
Jul 30 03:09:56.801261 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:56.801228    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab96c315-6328-41ed-8ccd-eecfd828f6d9" path="/var/lib/kubelet/pods/ab96c315-6328-41ed-8ccd-eecfd828f6d9/volumes"
Jul 30 03:09:58.225270 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.225229    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"]
Jul 30 03:09:58.225841 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.225821    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab96c315-6328-41ed-8ccd-eecfd828f6d9" containerName="istio-proxy"
Jul 30 03:09:58.225926 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.225846    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab96c315-6328-41ed-8ccd-eecfd828f6d9" containerName="istio-proxy"
Jul 30 03:09:58.226037 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.225952    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab96c315-6328-41ed-8ccd-eecfd828f6d9" containerName="istio-proxy"
Jul 30 03:09:58.230521 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.230500    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.239469 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.239449    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1a62a0-openshift-default-dockercfg-466w9\""
Jul 30 03:09:58.288623 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.288548    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"]
Jul 30 03:09:58.330904 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.330874    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-data\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.331080 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.330908    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istiod-ca-cert\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.331080 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.330976    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-workload-certs\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.331080 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.331003    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-token\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.331080 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.331044    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-workload-socket\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.331080 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.331070    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-podinfo\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.331290 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.331114    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-credential-socket\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.331290 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.331162    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-envoy\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.331290 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.331220    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2qxd\" (UniqueName: \"kubernetes.io/projected/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-kube-api-access-b2qxd\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.432496 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.432464    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-workload-certs\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.432699 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.432501    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-token\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.432699 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.432531    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-workload-socket\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.432699 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.432553    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-podinfo\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.432699 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.432571    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-credential-socket\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.432699 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.432611    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-envoy\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.432699 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.432634    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2qxd\" (UniqueName: \"kubernetes.io/projected/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-kube-api-access-b2qxd\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.432699 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.432652    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-data\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.432699 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.432676    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istiod-ca-cert\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.433108 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.432984    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-workload-certs\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.433108 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.433026    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-credential-socket\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.433272 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.433127    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-data\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.433326 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.433283    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-workload-socket\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.433623 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.433602    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istiod-ca-cert\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.435101 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.435079    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-envoy\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.435225 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.435209    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-podinfo\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.448526 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.448504    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-token\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.449791 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.449773    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2qxd\" (UniqueName: \"kubernetes.io/projected/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-kube-api-access-b2qxd\") pod \"e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") " pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.541099 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.541068    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:58.709566 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.709535    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"]
Jul 30 03:09:58.710343 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:09:58.710317    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebaa67f8_a72a_4a3a_b1b0_3d02eee3d7ac.slice/crio-2f56c31cb70909350352c7c7d5c299414067af38574882b962971ce29b1e8e32 WatchSource:0}: Error finding container 2f56c31cb70909350352c7c7d5c299414067af38574882b962971ce29b1e8e32: Status 404 returned error can't find the container with id 2f56c31cb70909350352c7c7d5c299414067af38574882b962971ce29b1e8e32
Jul 30 03:09:58.712497 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.712464    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:09:58.712604 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.712547    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:09:58.712604 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.712591    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:09:58.780768 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.780738    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv" event={"ID":"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac","Type":"ContainerStarted","Data":"7147d3b3a30e3af26a632b9a56af17e9dbc4e54858dbc0b86963145a28aca497"}
Jul 30 03:09:58.780880 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.780793    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv" event={"ID":"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac","Type":"ContainerStarted","Data":"2f56c31cb70909350352c7c7d5c299414067af38574882b962971ce29b1e8e32"}
Jul 30 03:09:58.846290 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:58.846152    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv" podStartSLOduration=0.846136632 podStartE2EDuration="846.136632ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:09:58.84500587 +0000 UTC m=+2634.491789278" watchObservedRunningTime="2026-07-30 03:09:58.846136632 +0000 UTC m=+2634.492920040"
Jul 30 03:09:59.541488 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:59.541450    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:59.546023 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:59.545998    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:59.786753 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:59.786717    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:09:59.787782 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:09:59.787756    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:10:00.766466 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:00.766440    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"
Jul 30 03:10:01.038349 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:01.038320    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"]
Jul 30 03:10:01.038986 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:01.038966    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab96c315-6328-41ed-8ccd-eecfd828f6d9" containerName="istio-proxy"
Jul 30 03:10:01.039053 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:01.038990    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab96c315-6328-41ed-8ccd-eecfd828f6d9" containerName="istio-proxy"
Jul 30 03:10:01.039101 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:01.039082    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab96c315-6328-41ed-8ccd-eecfd828f6d9" containerName="istio-proxy"
Jul 30 03:10:01.042726 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:01.042709    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"
Jul 30 03:10:01.050235 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:01.050212    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1a62a0-serving-cert\""
Jul 30 03:10:01.071019 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:01.070995    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"]
Jul 30 03:10:01.158430 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:01.158394    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44f1a243-4b96-4e77-ad0e-e5181900b38d-maas-api-tls\") pod \"maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn\" (UID: \"44f1a243-4b96-4e77-ad0e-e5181900b38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"
Jul 30 03:10:01.158430 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:01.158436    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vnrv\" (UniqueName: \"kubernetes.io/projected/44f1a243-4b96-4e77-ad0e-e5181900b38d-kube-api-access-7vnrv\") pod \"maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn\" (UID: \"44f1a243-4b96-4e77-ad0e-e5181900b38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"
Jul 30 03:10:01.158648 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:01.158460    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44f1a243-4b96-4e77-ad0e-e5181900b38d-openshift-service-ca\") pod \"maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn\" (UID: \"44f1a243-4b96-4e77-ad0e-e5181900b38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"
Jul 30 03:10:01.259569 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:01.259531    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44f1a243-4b96-4e77-ad0e-e5181900b38d-maas-api-tls\") pod \"maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn\" (UID: \"44f1a243-4b96-4e77-ad0e-e5181900b38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"
Jul 30 03:10:01.259772 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:01.259576    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vnrv\" (UniqueName: \"kubernetes.io/projected/44f1a243-4b96-4e77-ad0e-e5181900b38d-kube-api-access-7vnrv\") pod \"maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn\" (UID: \"44f1a243-4b96-4e77-ad0e-e5181900b38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"
Jul 30 03:10:01.259772 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:01.259611    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44f1a243-4b96-4e77-ad0e-e5181900b38d-openshift-service-ca\") pod \"maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn\" (UID: \"44f1a243-4b96-4e77-ad0e-e5181900b38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"
Jul 30 03:10:01.260507 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:01.260483    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44f1a243-4b96-4e77-ad0e-e5181900b38d-openshift-service-ca\") pod \"maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn\" (UID: \"44f1a243-4b96-4e77-ad0e-e5181900b38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"
Jul 30 03:10:01.262204 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:01.262150    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44f1a243-4b96-4e77-ad0e-e5181900b38d-maas-api-tls\") pod \"maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn\" (UID: \"44f1a243-4b96-4e77-ad0e-e5181900b38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"
Jul 30 03:10:01.276400 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:01.276373    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vnrv\" (UniqueName: \"kubernetes.io/projected/44f1a243-4b96-4e77-ad0e-e5181900b38d-kube-api-access-7vnrv\") pod \"maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn\" (UID: \"44f1a243-4b96-4e77-ad0e-e5181900b38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"
Jul 30 03:10:01.354205 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:01.354092    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"
Jul 30 03:10:01.501496 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:01.501470    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"]
Jul 30 03:10:01.504217 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:10:01.504189    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f1a243_4b96_4e77_ad0e_e5181900b38d.slice/crio-b1e457501b29e5797d1fea1134be8ff1c1817fdfa03b9df4f1be897aa36bdf22 WatchSource:0}: Error finding container b1e457501b29e5797d1fea1134be8ff1c1817fdfa03b9df4f1be897aa36bdf22: Status 404 returned error can't find the container with id b1e457501b29e5797d1fea1134be8ff1c1817fdfa03b9df4f1be897aa36bdf22
Jul 30 03:10:01.802137 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:01.802097    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn" event={"ID":"44f1a243-4b96-4e77-ad0e-e5181900b38d","Type":"ContainerStarted","Data":"b1e457501b29e5797d1fea1134be8ff1c1817fdfa03b9df4f1be897aa36bdf22"}
Jul 30 03:10:02.809601 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:02.809558    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn" event={"ID":"44f1a243-4b96-4e77-ad0e-e5181900b38d","Type":"ContainerStarted","Data":"438db35bef408f80bb33e765d38627c2c22e1624979877cec7a06cc073769cc9"}
Jul 30 03:10:02.809997 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:02.809633    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"
Jul 30 03:10:02.857780 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:02.857729    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn" podStartSLOduration=2.443045218 podStartE2EDuration="2.85771182s" totalImagesPullingTime="414.666602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:10:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:10:02.849072316 +0000 UTC m=+2638.495855699" watchObservedRunningTime="2026-07-30 03:10:02.85771182 +0000 UTC m=+2638.504495225"
Jul 30 03:10:05.763253 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:05.763228    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q"
Jul 30 03:10:07.770385 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:07.770344    2565 patch_prober.go:28] interesting pod/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 30 03:10:07.770854 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:07.770436    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2" podUID="8e09af48-cdfd-492c-83f6-0f18b7e7469f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 30 03:10:08.821601 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:08.821573    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"
Jul 30 03:10:09.854925 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:09.854892    2565 generic.go:356] "Generic (PLEG): container finished" podID="8e09af48-cdfd-492c-83f6-0f18b7e7469f" containerID="d590c4bca6f856017cbdd8ae29594d414c3327ca24de26f2fc84de87552a9b86" exitCode=0
Jul 30 03:10:09.855403 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:09.854956    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2" event={"ID":"8e09af48-cdfd-492c-83f6-0f18b7e7469f","Type":"ContainerDied","Data":"d590c4bca6f856017cbdd8ae29594d414c3327ca24de26f2fc84de87552a9b86"}
Jul 30 03:10:09.855496 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:09.855478    2565 scope.go:117] "RemoveContainer" containerID="d590c4bca6f856017cbdd8ae29594d414c3327ca24de26f2fc84de87552a9b86"
Jul 30 03:10:09.856712 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:09.856680    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:10:09.856805 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:09.856758    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:10:09.856805 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:09.856793    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:10:10.870361 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:10.870324    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2" event={"ID":"8e09af48-cdfd-492c-83f6-0f18b7e7469f","Type":"ContainerStarted","Data":"894267bd5eebfa1d6acbbf202851ffe0c2208cd8ec1d0d97eaab817b4ba4021b"}
Jul 30 03:10:11.177875 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:11.177776    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:10:11.178282 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:11.178157    2565 patch_prober.go:28] interesting pod/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Jul 30 03:10:11.178475 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:11.178450    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2" podUID="8e09af48-cdfd-492c-83f6-0f18b7e7469f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Jul 30 03:10:12.177875 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:12.177830    2565 patch_prober.go:28] interesting pod/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Jul 30 03:10:12.178432 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:12.177906    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2" podUID="8e09af48-cdfd-492c-83f6-0f18b7e7469f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Jul 30 03:10:13.178233 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:13.178189    2565 patch_prober.go:28] interesting pod/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Jul 30 03:10:13.178791 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:13.178267    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2" podUID="8e09af48-cdfd-492c-83f6-0f18b7e7469f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Jul 30 03:10:14.178078 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:14.178029    2565 patch_prober.go:28] interesting pod/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Jul 30 03:10:14.178320 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:14.178116    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2" podUID="8e09af48-cdfd-492c-83f6-0f18b7e7469f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Jul 30 03:10:15.185496 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:15.185459    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:10:15.187352 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:15.187332    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:10:15.188658 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:15.188641    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:10:18.931066 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:18.931030    2565 generic.go:356] "Generic (PLEG): container finished" podID="ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" containerID="7147d3b3a30e3af26a632b9a56af17e9dbc4e54858dbc0b86963145a28aca497" exitCode=0
Jul 30 03:10:18.931579 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:18.931102    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv" event={"ID":"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac","Type":"ContainerDied","Data":"7147d3b3a30e3af26a632b9a56af17e9dbc4e54858dbc0b86963145a28aca497"}
Jul 30 03:10:18.931579 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:18.931524    2565 scope.go:117] "RemoveContainer" containerID="7147d3b3a30e3af26a632b9a56af17e9dbc4e54858dbc0b86963145a28aca497"
Jul 30 03:10:18.932589 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:18.932559    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:10:18.932685 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:18.932627    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:10:18.932685 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:18.932660    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:10:19.941794 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:19.941757    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv" event={"ID":"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac","Type":"ContainerStarted","Data":"44311f5b6b39579473fb4c1a1357daa1949ca9ef1e870e80e4e858d58028db6e"}
Jul 30 03:10:20.541382 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:20.541346    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:10:20.541599 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:20.541562    2565 patch_prober.go:28] interesting pod/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Jul 30 03:10:20.541669 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:20.541616    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv" podUID="ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Jul 30 03:10:21.541558 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:21.541517    2565 patch_prober.go:28] interesting pod/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Jul 30 03:10:21.541957 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:21.541582    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv" podUID="ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Jul 30 03:10:22.541675 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:22.541630    2565 patch_prober.go:28] interesting pod/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Jul 30 03:10:22.542095 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:22.541704    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv" podUID="ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Jul 30 03:10:23.541737 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:23.541704    2565 patch_prober.go:28] interesting pod/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Jul 30 03:10:23.542228 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:23.541756    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv" podUID="ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Jul 30 03:10:24.545052 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:24.545023    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:10:24.545539 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:24.545315    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:10:24.546044 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:24.546028    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:10:28.970076 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:28.970037    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"]
Jul 30 03:10:28.970551 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:28.970279    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4" podUID="411c6990-f13c-436a-85b3-e5aa308421d1" containerName="maas-api" containerID="cri-o://28ad4bacc3eaee101f71cefefc0fe6c03c604b11f4bb1e2eb64d5752dc938afb" gracePeriod=30
Jul 30 03:10:29.119322 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:29.119279    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q"]
Jul 30 03:10:29.119573 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:29.119539    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q" podUID="53b57a2c-c367-41b8-b356-da81ec88306f" containerName="payload-processing" containerID="cri-o://aade64ad0332a3e586173ec67ea16152ed23e5d066e8668064b3ae7c7fdfc296" gracePeriod=30
Jul 30 03:10:29.248351 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:29.248325    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"
Jul 30 03:10:29.265268 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:29.265203    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411c6990-f13c-436a-85b3-e5aa308421d1-openshift-service-ca\") pod \"411c6990-f13c-436a-85b3-e5aa308421d1\" (UID: \"411c6990-f13c-436a-85b3-e5aa308421d1\") "
Jul 30 03:10:29.265456 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:29.265291    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411c6990-f13c-436a-85b3-e5aa308421d1-maas-api-tls\") pod \"411c6990-f13c-436a-85b3-e5aa308421d1\" (UID: \"411c6990-f13c-436a-85b3-e5aa308421d1\") "
Jul 30 03:10:29.265456 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:29.265368    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4k48w\" (UniqueName: \"kubernetes.io/projected/411c6990-f13c-436a-85b3-e5aa308421d1-kube-api-access-4k48w\") pod \"411c6990-f13c-436a-85b3-e5aa308421d1\" (UID: \"411c6990-f13c-436a-85b3-e5aa308421d1\") "
Jul 30 03:10:29.265698 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:29.265653    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/411c6990-f13c-436a-85b3-e5aa308421d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "411c6990-f13c-436a-85b3-e5aa308421d1" (UID: "411c6990-f13c-436a-85b3-e5aa308421d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:10:29.268088 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:29.268059    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411c6990-f13c-436a-85b3-e5aa308421d1-kube-api-access-4k48w" (OuterVolumeSpecName: "kube-api-access-4k48w") pod "411c6990-f13c-436a-85b3-e5aa308421d1" (UID: "411c6990-f13c-436a-85b3-e5aa308421d1"). InnerVolumeSpecName "kube-api-access-4k48w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:10:29.268333 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:29.268313    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/411c6990-f13c-436a-85b3-e5aa308421d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "411c6990-f13c-436a-85b3-e5aa308421d1" (UID: "411c6990-f13c-436a-85b3-e5aa308421d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:10:29.367143 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:29.367115    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k48w\" (UniqueName: \"kubernetes.io/projected/411c6990-f13c-436a-85b3-e5aa308421d1-kube-api-access-4k48w\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:10:29.367143 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:29.367144    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411c6990-f13c-436a-85b3-e5aa308421d1-openshift-service-ca\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:10:29.367343 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:29.367154    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411c6990-f13c-436a-85b3-e5aa308421d1-maas-api-tls\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:10:29.376905 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:29.376624    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q"
Jul 30 03:10:29.468408 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:29.468376    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53b57a2c-c367-41b8-b356-da81ec88306f-plugins-config-volume\") pod \"53b57a2c-c367-41b8-b356-da81ec88306f\" (UID: \"53b57a2c-c367-41b8-b356-da81ec88306f\") "
Jul 30 03:10:29.468611 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:29.468479    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbwz8\" (UniqueName: \"kubernetes.io/projected/53b57a2c-c367-41b8-b356-da81ec88306f-kube-api-access-rbwz8\") pod \"53b57a2c-c367-41b8-b356-da81ec88306f\" (UID: \"53b57a2c-c367-41b8-b356-da81ec88306f\") "
Jul 30 03:10:29.468787 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:29.468763    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b57a2c-c367-41b8-b356-da81ec88306f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "53b57a2c-c367-41b8-b356-da81ec88306f" (UID: "53b57a2c-c367-41b8-b356-da81ec88306f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:10:29.470686 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:29.470665    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b57a2c-c367-41b8-b356-da81ec88306f-kube-api-access-rbwz8" (OuterVolumeSpecName: "kube-api-access-rbwz8") pod "53b57a2c-c367-41b8-b356-da81ec88306f" (UID: "53b57a2c-c367-41b8-b356-da81ec88306f"). InnerVolumeSpecName "kube-api-access-rbwz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:10:29.569527 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:29.569495    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbwz8\" (UniqueName: \"kubernetes.io/projected/53b57a2c-c367-41b8-b356-da81ec88306f-kube-api-access-rbwz8\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:10:29.569527 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:29.569526    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53b57a2c-c367-41b8-b356-da81ec88306f-plugins-config-volume\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:10:30.001745 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:30.001712    2565 generic.go:356] "Generic (PLEG): container finished" podID="53b57a2c-c367-41b8-b356-da81ec88306f" containerID="aade64ad0332a3e586173ec67ea16152ed23e5d066e8668064b3ae7c7fdfc296" exitCode=0
Jul 30 03:10:30.002217 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:30.001764    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q"
Jul 30 03:10:30.002217 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:30.001802    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q" event={"ID":"53b57a2c-c367-41b8-b356-da81ec88306f","Type":"ContainerDied","Data":"aade64ad0332a3e586173ec67ea16152ed23e5d066e8668064b3ae7c7fdfc296"}
Jul 30 03:10:30.002217 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:30.001859    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q" event={"ID":"53b57a2c-c367-41b8-b356-da81ec88306f","Type":"ContainerDied","Data":"a11f522007cafbde692838f98794af22e3852eac11e1358da1ea2b6badbf82ca"}
Jul 30 03:10:30.002217 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:30.001882    2565 scope.go:117] "RemoveContainer" containerID="aade64ad0332a3e586173ec67ea16152ed23e5d066e8668064b3ae7c7fdfc296"
Jul 30 03:10:30.003459 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:30.003435    2565 generic.go:356] "Generic (PLEG): container finished" podID="411c6990-f13c-436a-85b3-e5aa308421d1" containerID="28ad4bacc3eaee101f71cefefc0fe6c03c604b11f4bb1e2eb64d5752dc938afb" exitCode=0
Jul 30 03:10:30.003561 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:30.003473    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4" event={"ID":"411c6990-f13c-436a-85b3-e5aa308421d1","Type":"ContainerDied","Data":"28ad4bacc3eaee101f71cefefc0fe6c03c604b11f4bb1e2eb64d5752dc938afb"}
Jul 30 03:10:30.003561 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:30.003478    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"
Jul 30 03:10:30.003561 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:30.003514    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4" event={"ID":"411c6990-f13c-436a-85b3-e5aa308421d1","Type":"ContainerDied","Data":"6ae88457178d86b913512d98b1ddc685f33e10df51a7f69e61321d2c325efdfe"}
Jul 30 03:10:30.013263 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:30.013148    2565 scope.go:117] "RemoveContainer" containerID="aade64ad0332a3e586173ec67ea16152ed23e5d066e8668064b3ae7c7fdfc296"
Jul 30 03:10:30.014070 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:10:30.013593    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aade64ad0332a3e586173ec67ea16152ed23e5d066e8668064b3ae7c7fdfc296\": container with ID starting with aade64ad0332a3e586173ec67ea16152ed23e5d066e8668064b3ae7c7fdfc296 not found: ID does not exist" containerID="aade64ad0332a3e586173ec67ea16152ed23e5d066e8668064b3ae7c7fdfc296"
Jul 30 03:10:30.014070 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:30.013659    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aade64ad0332a3e586173ec67ea16152ed23e5d066e8668064b3ae7c7fdfc296"} err="failed to get container status \"aade64ad0332a3e586173ec67ea16152ed23e5d066e8668064b3ae7c7fdfc296\": rpc error: code = NotFound desc = could not find container \"aade64ad0332a3e586173ec67ea16152ed23e5d066e8668064b3ae7c7fdfc296\": container with ID starting with aade64ad0332a3e586173ec67ea16152ed23e5d066e8668064b3ae7c7fdfc296 not found: ID does not exist"
Jul 30 03:10:30.014070 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:30.013679    2565 scope.go:117] "RemoveContainer" containerID="28ad4bacc3eaee101f71cefefc0fe6c03c604b11f4bb1e2eb64d5752dc938afb"
Jul 30 03:10:30.023854 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:30.023831    2565 scope.go:117] "RemoveContainer" containerID="28ad4bacc3eaee101f71cefefc0fe6c03c604b11f4bb1e2eb64d5752dc938afb"
Jul 30 03:10:30.024205 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:10:30.024155    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ad4bacc3eaee101f71cefefc0fe6c03c604b11f4bb1e2eb64d5752dc938afb\": container with ID starting with 28ad4bacc3eaee101f71cefefc0fe6c03c604b11f4bb1e2eb64d5752dc938afb not found: ID does not exist" containerID="28ad4bacc3eaee101f71cefefc0fe6c03c604b11f4bb1e2eb64d5752dc938afb"
Jul 30 03:10:30.024276 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:30.024213    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ad4bacc3eaee101f71cefefc0fe6c03c604b11f4bb1e2eb64d5752dc938afb"} err="failed to get container status \"28ad4bacc3eaee101f71cefefc0fe6c03c604b11f4bb1e2eb64d5752dc938afb\": rpc error: code = NotFound desc = could not find container \"28ad4bacc3eaee101f71cefefc0fe6c03c604b11f4bb1e2eb64d5752dc938afb\": container with ID starting with 28ad4bacc3eaee101f71cefefc0fe6c03c604b11f4bb1e2eb64d5752dc938afb not found: ID does not exist"
Jul 30 03:10:30.030483 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:30.030458    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q"]
Jul 30 03:10:30.036982 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:30.036954    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-970a97-6f596d9d4b-29q8q"]
Jul 30 03:10:30.051292 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:30.051260    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"]
Jul 30 03:10:30.062632 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:30.062606    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-970a97-c44dc9595-v89b4"]
Jul 30 03:10:30.800506 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:30.800468    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="411c6990-f13c-436a-85b3-e5aa308421d1" path="/var/lib/kubelet/pods/411c6990-f13c-436a-85b3-e5aa308421d1/volumes"
Jul 30 03:10:30.800887 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:30.800869    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53b57a2c-c367-41b8-b356-da81ec88306f" path="/var/lib/kubelet/pods/53b57a2c-c367-41b8-b356-da81ec88306f/volumes"
Jul 30 03:10:41.277759 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:41.277716    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"]
Jul 30 03:10:41.278372 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:41.278000    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2" podUID="8e09af48-cdfd-492c-83f6-0f18b7e7469f" containerName="istio-proxy" containerID="cri-o://894267bd5eebfa1d6acbbf202851ffe0c2208cd8ec1d0d97eaab817b4ba4021b" gracePeriod=30
Jul 30 03:10:45.189004 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:45.188964    2565 patch_prober.go:28] interesting pod/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 30 03:10:45.189505 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:45.189063    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2" podUID="8e09af48-cdfd-492c-83f6-0f18b7e7469f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 03:10:46.533743 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.533719    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:10:46.635645 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.635555    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-token\") pod \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") "
Jul 30 03:10:46.635645 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.635592    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-data\") pod \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") "
Jul 30 03:10:46.635645 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.635624    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6l4r5\" (UniqueName: \"kubernetes.io/projected/8e09af48-cdfd-492c-83f6-0f18b7e7469f-kube-api-access-6l4r5\") pod \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") "
Jul 30 03:10:46.635994 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.635650    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-workload-certs\") pod \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") "
Jul 30 03:10:46.635994 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.635676    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-credential-socket\") pod \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") "
Jul 30 03:10:46.635994 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.635707    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istiod-ca-cert\") pod \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") "
Jul 30 03:10:46.635994 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.635730    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-workload-socket\") pod \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") "
Jul 30 03:10:46.635994 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.635757    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-podinfo\") pod \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") "
Jul 30 03:10:46.635994 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.635791    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-envoy\") pod \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\" (UID: \"8e09af48-cdfd-492c-83f6-0f18b7e7469f\") "
Jul 30 03:10:46.635994 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.635942    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8e09af48-cdfd-492c-83f6-0f18b7e7469f" (UID: "8e09af48-cdfd-492c-83f6-0f18b7e7469f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:10:46.636644 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.636088    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-workload-certs\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:10:46.636644 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.636080    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8e09af48-cdfd-492c-83f6-0f18b7e7469f" (UID: "8e09af48-cdfd-492c-83f6-0f18b7e7469f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:10:46.636644 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.636340    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8e09af48-cdfd-492c-83f6-0f18b7e7469f" (UID: "8e09af48-cdfd-492c-83f6-0f18b7e7469f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:10:46.636786 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.636679    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8e09af48-cdfd-492c-83f6-0f18b7e7469f" (UID: "8e09af48-cdfd-492c-83f6-0f18b7e7469f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:10:46.636826 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.636788    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-data" (OuterVolumeSpecName: "istio-data") pod "8e09af48-cdfd-492c-83f6-0f18b7e7469f" (UID: "8e09af48-cdfd-492c-83f6-0f18b7e7469f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:10:46.637980 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.637956    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e09af48-cdfd-492c-83f6-0f18b7e7469f-kube-api-access-6l4r5" (OuterVolumeSpecName: "kube-api-access-6l4r5") pod "8e09af48-cdfd-492c-83f6-0f18b7e7469f" (UID: "8e09af48-cdfd-492c-83f6-0f18b7e7469f"). InnerVolumeSpecName "kube-api-access-6l4r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:10:46.638088 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.637998    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-token" (OuterVolumeSpecName: "istio-token") pod "8e09af48-cdfd-492c-83f6-0f18b7e7469f" (UID: "8e09af48-cdfd-492c-83f6-0f18b7e7469f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:10:46.638673 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.638646    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8e09af48-cdfd-492c-83f6-0f18b7e7469f" (UID: "8e09af48-cdfd-492c-83f6-0f18b7e7469f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:10:46.638673 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.638652    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8e09af48-cdfd-492c-83f6-0f18b7e7469f" (UID: "8e09af48-cdfd-492c-83f6-0f18b7e7469f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 30 03:10:46.736709 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.736662    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-data\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:10:46.736709 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.736704    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l4r5\" (UniqueName: \"kubernetes.io/projected/8e09af48-cdfd-492c-83f6-0f18b7e7469f-kube-api-access-6l4r5\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:10:46.736709 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.736715    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-credential-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:10:46.736709 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.736723    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istiod-ca-cert\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:10:46.736981 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.736733    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-workload-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:10:46.736981 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.736740    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-podinfo\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:10:46.736981 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.736748    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-envoy\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:10:46.736981 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:46.736755    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e09af48-cdfd-492c-83f6-0f18b7e7469f-istio-token\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:10:47.109188 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:47.109137    2565 generic.go:356] "Generic (PLEG): container finished" podID="8e09af48-cdfd-492c-83f6-0f18b7e7469f" containerID="894267bd5eebfa1d6acbbf202851ffe0c2208cd8ec1d0d97eaab817b4ba4021b" exitCode=0
Jul 30 03:10:47.109374 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:47.109209    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"
Jul 30 03:10:47.109374 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:47.109210    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2" event={"ID":"8e09af48-cdfd-492c-83f6-0f18b7e7469f","Type":"ContainerDied","Data":"894267bd5eebfa1d6acbbf202851ffe0c2208cd8ec1d0d97eaab817b4ba4021b"}
Jul 30 03:10:47.109374 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:47.109270    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2" event={"ID":"8e09af48-cdfd-492c-83f6-0f18b7e7469f","Type":"ContainerDied","Data":"a0f4de8ac507433f396f2993c2945d800a761ebe03cf7f2361c0405dee42cd66"}
Jul 30 03:10:47.109374 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:47.109292    2565 scope.go:117] "RemoveContainer" containerID="894267bd5eebfa1d6acbbf202851ffe0c2208cd8ec1d0d97eaab817b4ba4021b"
Jul 30 03:10:47.119007 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:47.118989    2565 scope.go:117] "RemoveContainer" containerID="d590c4bca6f856017cbdd8ae29594d414c3327ca24de26f2fc84de87552a9b86"
Jul 30 03:10:47.127626 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:47.127570    2565 scope.go:117] "RemoveContainer" containerID="894267bd5eebfa1d6acbbf202851ffe0c2208cd8ec1d0d97eaab817b4ba4021b"
Jul 30 03:10:47.127837 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:10:47.127806    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"894267bd5eebfa1d6acbbf202851ffe0c2208cd8ec1d0d97eaab817b4ba4021b\": container with ID starting with 894267bd5eebfa1d6acbbf202851ffe0c2208cd8ec1d0d97eaab817b4ba4021b not found: ID does not exist" containerID="894267bd5eebfa1d6acbbf202851ffe0c2208cd8ec1d0d97eaab817b4ba4021b"
Jul 30 03:10:47.127897 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:47.127846    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"894267bd5eebfa1d6acbbf202851ffe0c2208cd8ec1d0d97eaab817b4ba4021b"} err="failed to get container status \"894267bd5eebfa1d6acbbf202851ffe0c2208cd8ec1d0d97eaab817b4ba4021b\": rpc error: code = NotFound desc = could not find container \"894267bd5eebfa1d6acbbf202851ffe0c2208cd8ec1d0d97eaab817b4ba4021b\": container with ID starting with 894267bd5eebfa1d6acbbf202851ffe0c2208cd8ec1d0d97eaab817b4ba4021b not found: ID does not exist"
Jul 30 03:10:47.127897 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:47.127865    2565 scope.go:117] "RemoveContainer" containerID="d590c4bca6f856017cbdd8ae29594d414c3327ca24de26f2fc84de87552a9b86"
Jul 30 03:10:47.128152 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:10:47.128125    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d590c4bca6f856017cbdd8ae29594d414c3327ca24de26f2fc84de87552a9b86\": container with ID starting with d590c4bca6f856017cbdd8ae29594d414c3327ca24de26f2fc84de87552a9b86 not found: ID does not exist" containerID="d590c4bca6f856017cbdd8ae29594d414c3327ca24de26f2fc84de87552a9b86"
Jul 30 03:10:47.128270 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:47.128157    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d590c4bca6f856017cbdd8ae29594d414c3327ca24de26f2fc84de87552a9b86"} err="failed to get container status \"d590c4bca6f856017cbdd8ae29594d414c3327ca24de26f2fc84de87552a9b86\": rpc error: code = NotFound desc = could not find container \"d590c4bca6f856017cbdd8ae29594d414c3327ca24de26f2fc84de87552a9b86\": container with ID starting with d590c4bca6f856017cbdd8ae29594d414c3327ca24de26f2fc84de87552a9b86 not found: ID does not exist"
Jul 30 03:10:47.131633 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:47.131611    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"]
Jul 30 03:10:47.135413 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:47.135392    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-970a97-openshift-default-7bd765f546-4ncs2"]
Jul 30 03:10:48.801386 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:48.801354    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e09af48-cdfd-492c-83f6-0f18b7e7469f" path="/var/lib/kubelet/pods/8e09af48-cdfd-492c-83f6-0f18b7e7469f/volumes"
Jul 30 03:10:57.150079 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:57.149993    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"]
Jul 30 03:10:57.150823 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:57.150769    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn" podUID="44f1a243-4b96-4e77-ad0e-e5181900b38d" containerName="maas-api" containerID="cri-o://438db35bef408f80bb33e765d38627c2c22e1624979877cec7a06cc073769cc9" gracePeriod=30
Jul 30 03:10:57.416547 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:57.416479    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"
Jul 30 03:10:57.548001 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:57.547969    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44f1a243-4b96-4e77-ad0e-e5181900b38d-openshift-service-ca\") pod \"44f1a243-4b96-4e77-ad0e-e5181900b38d\" (UID: \"44f1a243-4b96-4e77-ad0e-e5181900b38d\") "
Jul 30 03:10:57.548203 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:57.548015    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44f1a243-4b96-4e77-ad0e-e5181900b38d-maas-api-tls\") pod \"44f1a243-4b96-4e77-ad0e-e5181900b38d\" (UID: \"44f1a243-4b96-4e77-ad0e-e5181900b38d\") "
Jul 30 03:10:57.548203 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:57.548075    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vnrv\" (UniqueName: \"kubernetes.io/projected/44f1a243-4b96-4e77-ad0e-e5181900b38d-kube-api-access-7vnrv\") pod \"44f1a243-4b96-4e77-ad0e-e5181900b38d\" (UID: \"44f1a243-4b96-4e77-ad0e-e5181900b38d\") "
Jul 30 03:10:57.548357 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:57.548335    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f1a243-4b96-4e77-ad0e-e5181900b38d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "44f1a243-4b96-4e77-ad0e-e5181900b38d" (UID: "44f1a243-4b96-4e77-ad0e-e5181900b38d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:10:57.550385 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:57.550357    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f1a243-4b96-4e77-ad0e-e5181900b38d-kube-api-access-7vnrv" (OuterVolumeSpecName: "kube-api-access-7vnrv") pod "44f1a243-4b96-4e77-ad0e-e5181900b38d" (UID: "44f1a243-4b96-4e77-ad0e-e5181900b38d"). InnerVolumeSpecName "kube-api-access-7vnrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:10:57.550491 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:57.550395    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44f1a243-4b96-4e77-ad0e-e5181900b38d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "44f1a243-4b96-4e77-ad0e-e5181900b38d" (UID: "44f1a243-4b96-4e77-ad0e-e5181900b38d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:10:57.649418 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:57.649382    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44f1a243-4b96-4e77-ad0e-e5181900b38d-maas-api-tls\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:10:57.649418 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:57.649411    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vnrv\" (UniqueName: \"kubernetes.io/projected/44f1a243-4b96-4e77-ad0e-e5181900b38d-kube-api-access-7vnrv\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:10:57.649418 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:57.649420    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44f1a243-4b96-4e77-ad0e-e5181900b38d-openshift-service-ca\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:10:58.183298 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:58.183259    2565 generic.go:356] "Generic (PLEG): container finished" podID="44f1a243-4b96-4e77-ad0e-e5181900b38d" containerID="438db35bef408f80bb33e765d38627c2c22e1624979877cec7a06cc073769cc9" exitCode=0
Jul 30 03:10:58.183774 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:58.183303    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"
Jul 30 03:10:58.183774 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:58.183348    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn" event={"ID":"44f1a243-4b96-4e77-ad0e-e5181900b38d","Type":"ContainerDied","Data":"438db35bef408f80bb33e765d38627c2c22e1624979877cec7a06cc073769cc9"}
Jul 30 03:10:58.183774 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:58.183401    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn" event={"ID":"44f1a243-4b96-4e77-ad0e-e5181900b38d","Type":"ContainerDied","Data":"b1e457501b29e5797d1fea1134be8ff1c1817fdfa03b9df4f1be897aa36bdf22"}
Jul 30 03:10:58.183774 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:58.183425    2565 scope.go:117] "RemoveContainer" containerID="438db35bef408f80bb33e765d38627c2c22e1624979877cec7a06cc073769cc9"
Jul 30 03:10:58.199537 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:58.199514    2565 scope.go:117] "RemoveContainer" containerID="438db35bef408f80bb33e765d38627c2c22e1624979877cec7a06cc073769cc9"
Jul 30 03:10:58.199837 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:10:58.199818    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"438db35bef408f80bb33e765d38627c2c22e1624979877cec7a06cc073769cc9\": container with ID starting with 438db35bef408f80bb33e765d38627c2c22e1624979877cec7a06cc073769cc9 not found: ID does not exist" containerID="438db35bef408f80bb33e765d38627c2c22e1624979877cec7a06cc073769cc9"
Jul 30 03:10:58.199894 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:58.199846    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"438db35bef408f80bb33e765d38627c2c22e1624979877cec7a06cc073769cc9"} err="failed to get container status \"438db35bef408f80bb33e765d38627c2c22e1624979877cec7a06cc073769cc9\": rpc error: code = NotFound desc = could not find container \"438db35bef408f80bb33e765d38627c2c22e1624979877cec7a06cc073769cc9\": container with ID starting with 438db35bef408f80bb33e765d38627c2c22e1624979877cec7a06cc073769cc9 not found: ID does not exist"
Jul 30 03:10:58.209624 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:58.209599    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"]
Jul 30 03:10:58.214674 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:58.214653    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1a62a0-6b49f68884-mpfqn"]
Jul 30 03:10:58.800795 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:10:58.800764    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44f1a243-4b96-4e77-ad0e-e5181900b38d" path="/var/lib/kubelet/pods/44f1a243-4b96-4e77-ad0e-e5181900b38d/volumes"
Jul 30 03:11:08.366930 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:08.366880    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"]
Jul 30 03:11:08.367425 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:08.367162    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv" podUID="ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" containerName="istio-proxy" containerID="cri-o://44311f5b6b39579473fb4c1a1357daa1949ca9ef1e870e80e4e858d58028db6e" gracePeriod=30
Jul 30 03:11:09.243691 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.243655    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"]
Jul 30 03:11:09.244091 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.244070    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="411c6990-f13c-436a-85b3-e5aa308421d1" containerName="maas-api"
Jul 30 03:11:09.244091 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.244091    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="411c6990-f13c-436a-85b3-e5aa308421d1" containerName="maas-api"
Jul 30 03:11:09.244408 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.244114    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44f1a243-4b96-4e77-ad0e-e5181900b38d" containerName="maas-api"
Jul 30 03:11:09.244408 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.244124    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="44f1a243-4b96-4e77-ad0e-e5181900b38d" containerName="maas-api"
Jul 30 03:11:09.244408 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.244150    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e09af48-cdfd-492c-83f6-0f18b7e7469f" containerName="istio-proxy"
Jul 30 03:11:09.244408 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.244160    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e09af48-cdfd-492c-83f6-0f18b7e7469f" containerName="istio-proxy"
Jul 30 03:11:09.244408 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.244189    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53b57a2c-c367-41b8-b356-da81ec88306f" containerName="payload-processing"
Jul 30 03:11:09.244408 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.244198    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="53b57a2c-c367-41b8-b356-da81ec88306f" containerName="payload-processing"
Jul 30 03:11:09.244408 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.244211    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e09af48-cdfd-492c-83f6-0f18b7e7469f" containerName="istio-proxy"
Jul 30 03:11:09.244408 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.244219    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e09af48-cdfd-492c-83f6-0f18b7e7469f" containerName="istio-proxy"
Jul 30 03:11:09.244408 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.244322    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="53b57a2c-c367-41b8-b356-da81ec88306f" containerName="payload-processing"
Jul 30 03:11:09.244408 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.244337    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="44f1a243-4b96-4e77-ad0e-e5181900b38d" containerName="maas-api"
Jul 30 03:11:09.244408 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.244351    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e09af48-cdfd-492c-83f6-0f18b7e7469f" containerName="istio-proxy"
Jul 30 03:11:09.244408 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.244360    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="411c6990-f13c-436a-85b3-e5aa308421d1" containerName="maas-api"
Jul 30 03:11:09.244408 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.244370    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e09af48-cdfd-492c-83f6-0f18b7e7469f" containerName="istio-proxy"
Jul 30 03:11:09.249340 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.249321    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.252454 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.252435    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-da5459-openshift-default-dockercfg-88svb\""
Jul 30 03:11:09.265707 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.265678    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"]
Jul 30 03:11:09.349499 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.349450    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-data\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.349699 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.349519    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba45532-1c80-4328-b6d2-c73ff607f539-istiod-ca-cert\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.349699 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.349553    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7vql\" (UniqueName: \"kubernetes.io/projected/8ba45532-1c80-4328-b6d2-c73ff607f539-kube-api-access-c7vql\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.349699 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.349596    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-workload-certs\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.349699 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.349627    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-envoy\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.349699 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.349688    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-token\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.349984 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.349716    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-workload-socket\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.349984 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.349757    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-credential-socket\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.349984 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.349794    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-podinfo\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.451012 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.450975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-podinfo\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.451536 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.451026    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-data\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.451536 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.451062    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba45532-1c80-4328-b6d2-c73ff607f539-istiod-ca-cert\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.451536 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.451089    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7vql\" (UniqueName: \"kubernetes.io/projected/8ba45532-1c80-4328-b6d2-c73ff607f539-kube-api-access-c7vql\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.451536 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.451131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-workload-certs\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.451536 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.451163    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-envoy\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.451536 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.451255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-token\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.451536 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.451285    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-workload-socket\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.451536 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.451336    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-credential-socket\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.451536 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.451505    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-data\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.451977 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.451570    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-workload-certs\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.451977 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.451682    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-credential-socket\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.451977 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.451816    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-workload-socket\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.451977 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.451886    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba45532-1c80-4328-b6d2-c73ff607f539-istiod-ca-cert\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.453662 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.453643    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-envoy\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.453791 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.453776    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-podinfo\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.460297 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.460276    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7vql\" (UniqueName: \"kubernetes.io/projected/8ba45532-1c80-4328-b6d2-c73ff607f539-kube-api-access-c7vql\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.460448 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.460428    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-token\") pod \"e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") " pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.548067 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.548029    2565 patch_prober.go:28] interesting pod/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 30 03:11:09.548253 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.548109    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv" podUID="ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 03:11:09.559358 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.559327    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:09.709041 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.709008    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"]
Jul 30 03:11:09.709221 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:11:09.709187    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba45532_1c80_4328_b6d2_c73ff607f539.slice/crio-207ab3d76bf467f064d1f10e4ebb8775535d0662374d3afc5e0f2e5e9531a27d WatchSource:0}: Error finding container 207ab3d76bf467f064d1f10e4ebb8775535d0662374d3afc5e0f2e5e9531a27d: Status 404 returned error can't find the container with id 207ab3d76bf467f064d1f10e4ebb8775535d0662374d3afc5e0f2e5e9531a27d
Jul 30 03:11:09.711508 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.711465    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:11:09.711604 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.711557    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:11:09.711647 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:09.711598    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:11:10.255125 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:10.255088    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p" event={"ID":"8ba45532-1c80-4328-b6d2-c73ff607f539","Type":"ContainerStarted","Data":"cb46a97547962b06b315845a17164e37c1a11319b71f6e39c820d66313bf79f5"}
Jul 30 03:11:10.255337 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:10.255141    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p" event={"ID":"8ba45532-1c80-4328-b6d2-c73ff607f539","Type":"ContainerStarted","Data":"207ab3d76bf467f064d1f10e4ebb8775535d0662374d3afc5e0f2e5e9531a27d"}
Jul 30 03:11:10.292408 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:10.292352    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p" podStartSLOduration=1.292334844 podStartE2EDuration="1.292334844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:11:10.28938523 +0000 UTC m=+2705.936168672" watchObservedRunningTime="2026-07-30 03:11:10.292334844 +0000 UTC m=+2705.939118248"
Jul 30 03:11:10.560077 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:10.560047    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:10.564863 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:10.564838    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:11.260976 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:11.260939    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:11.262031 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:11.262013    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:12.540380 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:12.540346    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"]
Jul 30 03:11:12.544252 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:12.544230    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"
Jul 30 03:11:12.547102 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:12.547079    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-da5459-serving-cert\""
Jul 30 03:11:12.557255 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:12.557227    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"]
Jul 30 03:11:12.585160 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:12.585122    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fb5b4d-5973-4860-bbf3-854e36472540-maas-api-tls\") pod \"maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms\" (UID: \"d4fb5b4d-5973-4860-bbf3-854e36472540\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"
Jul 30 03:11:12.585351 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:12.585204    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpj8q\" (UniqueName: \"kubernetes.io/projected/d4fb5b4d-5973-4860-bbf3-854e36472540-kube-api-access-vpj8q\") pod \"maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms\" (UID: \"d4fb5b4d-5973-4860-bbf3-854e36472540\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"
Jul 30 03:11:12.585351 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:12.585232    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fb5b4d-5973-4860-bbf3-854e36472540-openshift-service-ca\") pod \"maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms\" (UID: \"d4fb5b4d-5973-4860-bbf3-854e36472540\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"
Jul 30 03:11:12.685689 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:12.685654    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpj8q\" (UniqueName: \"kubernetes.io/projected/d4fb5b4d-5973-4860-bbf3-854e36472540-kube-api-access-vpj8q\") pod \"maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms\" (UID: \"d4fb5b4d-5973-4860-bbf3-854e36472540\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"
Jul 30 03:11:12.685918 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:12.685708    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fb5b4d-5973-4860-bbf3-854e36472540-openshift-service-ca\") pod \"maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms\" (UID: \"d4fb5b4d-5973-4860-bbf3-854e36472540\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"
Jul 30 03:11:12.685918 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:12.685825    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fb5b4d-5973-4860-bbf3-854e36472540-maas-api-tls\") pod \"maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms\" (UID: \"d4fb5b4d-5973-4860-bbf3-854e36472540\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"
Jul 30 03:11:12.686042 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:11:12.685957    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-serving-cert: secret "maas-api-e2e-shared-a-da5459-serving-cert" not found
Jul 30 03:11:12.686042 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:11:12.686027    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4fb5b4d-5973-4860-bbf3-854e36472540-maas-api-tls podName:d4fb5b4d-5973-4860-bbf3-854e36472540 nodeName:}" failed. No retries permitted until 2026-07-30 03:11:13.186005333 +0000 UTC m=+2708.832788738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d4fb5b4d-5973-4860-bbf3-854e36472540-maas-api-tls") pod "maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms" (UID: "d4fb5b4d-5973-4860-bbf3-854e36472540") : secret "maas-api-e2e-shared-a-da5459-serving-cert" not found
Jul 30 03:11:12.686577 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:12.686552    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fb5b4d-5973-4860-bbf3-854e36472540-openshift-service-ca\") pod \"maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms\" (UID: \"d4fb5b4d-5973-4860-bbf3-854e36472540\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"
Jul 30 03:11:12.697001 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:12.696963    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpj8q\" (UniqueName: \"kubernetes.io/projected/d4fb5b4d-5973-4860-bbf3-854e36472540-kube-api-access-vpj8q\") pod \"maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms\" (UID: \"d4fb5b4d-5973-4860-bbf3-854e36472540\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"
Jul 30 03:11:13.192403 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.192365    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fb5b4d-5973-4860-bbf3-854e36472540-maas-api-tls\") pod \"maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms\" (UID: \"d4fb5b4d-5973-4860-bbf3-854e36472540\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"
Jul 30 03:11:13.194968 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.194943    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fb5b4d-5973-4860-bbf3-854e36472540-maas-api-tls\") pod \"maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms\" (UID: \"d4fb5b4d-5973-4860-bbf3-854e36472540\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"
Jul 30 03:11:13.456391 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.456296    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"
Jul 30 03:11:13.635590 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.635559    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:11:13.642485 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.642464    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"]
Jul 30 03:11:13.643342 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:11:13.643314    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fb5b4d_5973_4860_bbf3_854e36472540.slice/crio-009fa4d05e4627b761842c3f04c7eb71cf6811319d1e2cc6db08e42c3fdbd146 WatchSource:0}: Error finding container 009fa4d05e4627b761842c3f04c7eb71cf6811319d1e2cc6db08e42c3fdbd146: Status 404 returned error can't find the container with id 009fa4d05e4627b761842c3f04c7eb71cf6811319d1e2cc6db08e42c3fdbd146
Jul 30 03:11:13.697142 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.697098    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2qxd\" (UniqueName: \"kubernetes.io/projected/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-kube-api-access-b2qxd\") pod \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") "
Jul 30 03:11:13.697320 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.697206    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istiod-ca-cert\") pod \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") "
Jul 30 03:11:13.697320 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.697241    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-workload-socket\") pod \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") "
Jul 30 03:11:13.697320 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.697275    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-credential-socket\") pod \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") "
Jul 30 03:11:13.697320 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.697305    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-data\") pod \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") "
Jul 30 03:11:13.697541 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.697364    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-workload-certs\") pod \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") "
Jul 30 03:11:13.697541 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.697387    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-envoy\") pod \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") "
Jul 30 03:11:13.697541 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.697440    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-token\") pod \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") "
Jul 30 03:11:13.697541 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.697468    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-podinfo\") pod \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\" (UID: \"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac\") "
Jul 30 03:11:13.697734 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.697566    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" (UID: "ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:11:13.697734 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.697596    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" (UID: "ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:11:13.697734 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.697691    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" (UID: "ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:11:13.697898 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.697824    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" (UID: "ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:11:13.697991 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.697968    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istiod-ca-cert\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:11:13.698061 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.697994    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-workload-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:11:13.698061 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.698008    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-credential-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:11:13.698061 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.698022    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-workload-certs\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:11:13.698628 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.698604    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-data" (OuterVolumeSpecName: "istio-data") pod "ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" (UID: "ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:11:13.699921 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.699902    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-token" (OuterVolumeSpecName: "istio-token") pod "ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" (UID: "ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:11:13.700161 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.700137    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" (UID: "ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 30 03:11:13.700161 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.700156    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-kube-api-access-b2qxd" (OuterVolumeSpecName: "kube-api-access-b2qxd") pod "ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" (UID: "ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac"). InnerVolumeSpecName "kube-api-access-b2qxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:11:13.700330 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.700219    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" (UID: "ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:11:13.798774 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.798737    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-envoy\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:11:13.798774 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.798764    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-token\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:11:13.798774 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.798773    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-podinfo\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:11:13.798774 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.798781    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2qxd\" (UniqueName: \"kubernetes.io/projected/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-kube-api-access-b2qxd\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:11:13.799039 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:13.798790    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac-istio-data\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:11:14.282184 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:14.282063    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms" event={"ID":"d4fb5b4d-5973-4860-bbf3-854e36472540","Type":"ContainerStarted","Data":"009fa4d05e4627b761842c3f04c7eb71cf6811319d1e2cc6db08e42c3fdbd146"}
Jul 30 03:11:14.284144 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:14.284113    2565 generic.go:356] "Generic (PLEG): container finished" podID="ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" containerID="44311f5b6b39579473fb4c1a1357daa1949ca9ef1e870e80e4e858d58028db6e" exitCode=0
Jul 30 03:11:14.284305 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:14.284200    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"
Jul 30 03:11:14.284305 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:14.284207    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv" event={"ID":"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac","Type":"ContainerDied","Data":"44311f5b6b39579473fb4c1a1357daa1949ca9ef1e870e80e4e858d58028db6e"}
Jul 30 03:11:14.284305 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:14.284268    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv" event={"ID":"ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac","Type":"ContainerDied","Data":"2f56c31cb70909350352c7c7d5c299414067af38574882b962971ce29b1e8e32"}
Jul 30 03:11:14.284305 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:14.284299    2565 scope.go:117] "RemoveContainer" containerID="44311f5b6b39579473fb4c1a1357daa1949ca9ef1e870e80e4e858d58028db6e"
Jul 30 03:11:14.295007 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:14.294987    2565 scope.go:117] "RemoveContainer" containerID="7147d3b3a30e3af26a632b9a56af17e9dbc4e54858dbc0b86963145a28aca497"
Jul 30 03:11:14.309053 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:14.309029    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"]
Jul 30 03:11:14.311095 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:14.311067    2565 scope.go:117] "RemoveContainer" containerID="44311f5b6b39579473fb4c1a1357daa1949ca9ef1e870e80e4e858d58028db6e"
Jul 30 03:11:14.311447 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:11:14.311415    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44311f5b6b39579473fb4c1a1357daa1949ca9ef1e870e80e4e858d58028db6e\": container with ID starting with 44311f5b6b39579473fb4c1a1357daa1949ca9ef1e870e80e4e858d58028db6e not found: ID does not exist" containerID="44311f5b6b39579473fb4c1a1357daa1949ca9ef1e870e80e4e858d58028db6e"
Jul 30 03:11:14.311536 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:14.311462    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44311f5b6b39579473fb4c1a1357daa1949ca9ef1e870e80e4e858d58028db6e"} err="failed to get container status \"44311f5b6b39579473fb4c1a1357daa1949ca9ef1e870e80e4e858d58028db6e\": rpc error: code = NotFound desc = could not find container \"44311f5b6b39579473fb4c1a1357daa1949ca9ef1e870e80e4e858d58028db6e\": container with ID starting with 44311f5b6b39579473fb4c1a1357daa1949ca9ef1e870e80e4e858d58028db6e not found: ID does not exist"
Jul 30 03:11:14.311536 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:14.311484    2565 scope.go:117] "RemoveContainer" containerID="7147d3b3a30e3af26a632b9a56af17e9dbc4e54858dbc0b86963145a28aca497"
Jul 30 03:11:14.311777 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:11:14.311755    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7147d3b3a30e3af26a632b9a56af17e9dbc4e54858dbc0b86963145a28aca497\": container with ID starting with 7147d3b3a30e3af26a632b9a56af17e9dbc4e54858dbc0b86963145a28aca497 not found: ID does not exist" containerID="7147d3b3a30e3af26a632b9a56af17e9dbc4e54858dbc0b86963145a28aca497"
Jul 30 03:11:14.311836 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:14.311783    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7147d3b3a30e3af26a632b9a56af17e9dbc4e54858dbc0b86963145a28aca497"} err="failed to get container status \"7147d3b3a30e3af26a632b9a56af17e9dbc4e54858dbc0b86963145a28aca497\": rpc error: code = NotFound desc = could not find container \"7147d3b3a30e3af26a632b9a56af17e9dbc4e54858dbc0b86963145a28aca497\": container with ID starting with 7147d3b3a30e3af26a632b9a56af17e9dbc4e54858dbc0b86963145a28aca497 not found: ID does not exist"
Jul 30 03:11:14.313581 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:14.313561    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1a62a0-openshift-default-7f5b9f6688-ct5wv"]
Jul 30 03:11:14.802906 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:14.802868    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" path="/var/lib/kubelet/pods/ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac/volumes"
Jul 30 03:11:15.292242 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:15.292210    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms" event={"ID":"d4fb5b4d-5973-4860-bbf3-854e36472540","Type":"ContainerStarted","Data":"3d3296000e1c82907aa856cdbfa804519f3021c6aea16c2cf6ff67824b559a66"}
Jul 30 03:11:15.292434 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:15.292382    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"
Jul 30 03:11:15.316108 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:15.316049    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms" podStartSLOduration=2.6415706820000002 podStartE2EDuration="3.316032909s" totalImagesPullingTime="674.462227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:11:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:11:15.315148896 +0000 UTC m=+2710.961932303" watchObservedRunningTime="2026-07-30 03:11:15.316032909 +0000 UTC m=+2710.962816355"
Jul 30 03:11:17.334308 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.334278    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"]
Jul 30 03:11:17.334710 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.334661    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" containerName="istio-proxy"
Jul 30 03:11:17.334710 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.334674    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" containerName="istio-proxy"
Jul 30 03:11:17.334710 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.334683    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" containerName="istio-proxy"
Jul 30 03:11:17.334710 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.334688    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" containerName="istio-proxy"
Jul 30 03:11:17.334867 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.334755    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" containerName="istio-proxy"
Jul 30 03:11:17.334867 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.334763    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebaa67f8-a72a-4a3a-b1b0-3d02eee3d7ac" containerName="istio-proxy"
Jul 30 03:11:17.338214 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.338194    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.341065 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.341043    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-f2a805-openshift-default-dockercfg-vqc97\""
Jul 30 03:11:17.351449 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.351429    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"]
Jul 30 03:11:17.432815 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.432761    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-credential-socket\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.433012 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.432864    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-data\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.433012 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.432901    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpgpc\" (UniqueName: \"kubernetes.io/projected/02bb64dc-de3e-4e34-b139-d6510f2eb893-kube-api-access-cpgpc\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.433012 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.432930    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-envoy\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.433012 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.432978    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-token\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.433249 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.433021    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02bb64dc-de3e-4e34-b139-d6510f2eb893-istiod-ca-cert\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.433249 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.433056    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-workload-socket\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.433249 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.433092    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-workload-certs\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.433249 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.433124    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-podinfo\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.534509 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.534475    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-credential-socket\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.534687 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.534538    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-data\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.534687 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.534561    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpgpc\" (UniqueName: \"kubernetes.io/projected/02bb64dc-de3e-4e34-b139-d6510f2eb893-kube-api-access-cpgpc\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.534687 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.534578    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-envoy\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.534687 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.534609    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-token\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.534687 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.534643    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02bb64dc-de3e-4e34-b139-d6510f2eb893-istiod-ca-cert\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.534687 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.534676    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-workload-socket\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.535066 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.534708    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-workload-certs\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.535066 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.534731    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-podinfo\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.535066 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.534941    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-credential-socket\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.535066 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.535039    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-data\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.535298 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.535283    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-workload-socket\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.535354 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.535287    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-workload-certs\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.535591 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.535566    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02bb64dc-de3e-4e34-b139-d6510f2eb893-istiod-ca-cert\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.537056 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.537038    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-envoy\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.537417 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.537395    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-podinfo\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.545750 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.545727    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpgpc\" (UniqueName: \"kubernetes.io/projected/02bb64dc-de3e-4e34-b139-d6510f2eb893-kube-api-access-cpgpc\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.545856 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.545795    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-token\") pod \"e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") " pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.648632 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.648536    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:17.793345 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.793313    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"]
Jul 30 03:11:17.794011 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:11:17.793987    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bb64dc_de3e_4e34_b139_d6510f2eb893.slice/crio-b0b25d1ab0a74263acbfcab9a7a9329f2ad9dd74c7141b27d728b66fc9acdbcf WatchSource:0}: Error finding container b0b25d1ab0a74263acbfcab9a7a9329f2ad9dd74c7141b27d728b66fc9acdbcf: Status 404 returned error can't find the container with id b0b25d1ab0a74263acbfcab9a7a9329f2ad9dd74c7141b27d728b66fc9acdbcf
Jul 30 03:11:17.796676 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.796645    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:11:17.796790 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.796722    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:11:17.796790 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:17.796768    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:11:18.313825 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:18.313785    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5" event={"ID":"02bb64dc-de3e-4e34-b139-d6510f2eb893","Type":"ContainerStarted","Data":"1866844b5c2fd1b790ec9cb765f9666484a08bfcc6345763495085fcab2bb331"}
Jul 30 03:11:18.314026 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:18.313838    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5" event={"ID":"02bb64dc-de3e-4e34-b139-d6510f2eb893","Type":"ContainerStarted","Data":"b0b25d1ab0a74263acbfcab9a7a9329f2ad9dd74c7141b27d728b66fc9acdbcf"}
Jul 30 03:11:18.348694 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:18.348637    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5" podStartSLOduration=1.34862111 podStartE2EDuration="1.34862111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:11:18.347385078 +0000 UTC m=+2713.994168493" watchObservedRunningTime="2026-07-30 03:11:18.34862111 +0000 UTC m=+2713.995404578"
Jul 30 03:11:18.649092 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:18.648995    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:18.653975 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:18.653950    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:19.319976 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:19.319944    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:19.320919 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:19.320900    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:20.520367 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:20.520334    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"]
Jul 30 03:11:20.524103 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:20.524086    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"
Jul 30 03:11:20.527735 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:20.527716    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-f2a805-serving-cert\""
Jul 30 03:11:20.554356 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:20.554321    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"]
Jul 30 03:11:20.566306 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:20.566273    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-maas-api-tls\") pod \"maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh\" (UID: \"990d6ed8-feb1-455b-ba3a-a5243bbd63fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"
Jul 30 03:11:20.566541 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:20.566323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh\" (UID: \"990d6ed8-feb1-455b-ba3a-a5243bbd63fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"
Jul 30 03:11:20.566541 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:20.566357    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xm6r\" (UniqueName: \"kubernetes.io/projected/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-kube-api-access-7xm6r\") pod \"maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh\" (UID: \"990d6ed8-feb1-455b-ba3a-a5243bbd63fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"
Jul 30 03:11:20.667261 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:20.667226    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xm6r\" (UniqueName: \"kubernetes.io/projected/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-kube-api-access-7xm6r\") pod \"maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh\" (UID: \"990d6ed8-feb1-455b-ba3a-a5243bbd63fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"
Jul 30 03:11:20.667436 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:20.667336    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-maas-api-tls\") pod \"maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh\" (UID: \"990d6ed8-feb1-455b-ba3a-a5243bbd63fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"
Jul 30 03:11:20.667436 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:20.667375    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh\" (UID: \"990d6ed8-feb1-455b-ba3a-a5243bbd63fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"
Jul 30 03:11:20.667580 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:11:20.667475    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-serving-cert: secret "maas-api-e2e-shared-b-f2a805-serving-cert" not found
Jul 30 03:11:20.667580 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:11:20.667543    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-maas-api-tls podName:990d6ed8-feb1-455b-ba3a-a5243bbd63fc nodeName:}" failed. No retries permitted until 2026-07-30 03:11:21.167525931 +0000 UTC m=+2716.814309320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-maas-api-tls") pod "maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh" (UID: "990d6ed8-feb1-455b-ba3a-a5243bbd63fc") : secret "maas-api-e2e-shared-b-f2a805-serving-cert" not found
Jul 30 03:11:20.667984 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:20.667965    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh\" (UID: \"990d6ed8-feb1-455b-ba3a-a5243bbd63fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"
Jul 30 03:11:20.703698 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:20.703670    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xm6r\" (UniqueName: \"kubernetes.io/projected/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-kube-api-access-7xm6r\") pod \"maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh\" (UID: \"990d6ed8-feb1-455b-ba3a-a5243bbd63fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"
Jul 30 03:11:21.172540 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:21.172471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-maas-api-tls\") pod \"maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh\" (UID: \"990d6ed8-feb1-455b-ba3a-a5243bbd63fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"
Jul 30 03:11:21.186815 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:21.186765    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-maas-api-tls\") pod \"maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh\" (UID: \"990d6ed8-feb1-455b-ba3a-a5243bbd63fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"
Jul 30 03:11:21.318652 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:21.318618    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"
Jul 30 03:11:21.434590 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:21.434490    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"
Jul 30 03:11:21.633742 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:21.633708    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"]
Jul 30 03:11:21.634364 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:11:21.634319    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990d6ed8_feb1_455b_ba3a_a5243bbd63fc.slice/crio-7c2443ef636550934af4cf65d8f9bff75363f1ba2f9a57f0dacbe4d7b39cd6cd WatchSource:0}: Error finding container 7c2443ef636550934af4cf65d8f9bff75363f1ba2f9a57f0dacbe4d7b39cd6cd: Status 404 returned error can't find the container with id 7c2443ef636550934af4cf65d8f9bff75363f1ba2f9a57f0dacbe4d7b39cd6cd
Jul 30 03:11:22.340028 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:22.339985    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh" event={"ID":"990d6ed8-feb1-455b-ba3a-a5243bbd63fc","Type":"ContainerStarted","Data":"0e7e445c62680b0d5158a87a2947e6a831cd02d53c3b4133262d63feb7fe6f4c"}
Jul 30 03:11:22.340257 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:22.340050    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh" event={"ID":"990d6ed8-feb1-455b-ba3a-a5243bbd63fc","Type":"ContainerStarted","Data":"7c2443ef636550934af4cf65d8f9bff75363f1ba2f9a57f0dacbe4d7b39cd6cd"}
Jul 30 03:11:22.340257 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:22.340079    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"
Jul 30 03:11:22.362685 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:22.362627    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh" podStartSLOduration=2.008473253 podStartE2EDuration="2.362605084s" totalImagesPullingTime="354.131831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:11:22.359957224 +0000 UTC m=+2718.006740631" watchObservedRunningTime="2026-07-30 03:11:22.362605084 +0000 UTC m=+2718.009388488"
Jul 30 03:11:27.302355 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:27.302310    2565 patch_prober.go:28] interesting pod/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 30 03:11:27.302764 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:27.302380    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p" podUID="8ba45532-1c80-4328-b6d2-c73ff607f539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 30 03:11:28.350879 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:28.350851    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"
Jul 30 03:11:28.376451 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:28.376420    2565 generic.go:356] "Generic (PLEG): container finished" podID="8ba45532-1c80-4328-b6d2-c73ff607f539" containerID="cb46a97547962b06b315845a17164e37c1a11319b71f6e39c820d66313bf79f5" exitCode=0
Jul 30 03:11:28.376640 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:28.376507    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p" event={"ID":"8ba45532-1c80-4328-b6d2-c73ff607f539","Type":"ContainerDied","Data":"cb46a97547962b06b315845a17164e37c1a11319b71f6e39c820d66313bf79f5"}
Jul 30 03:11:28.376919 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:28.376905    2565 scope.go:117] "RemoveContainer" containerID="cb46a97547962b06b315845a17164e37c1a11319b71f6e39c820d66313bf79f5"
Jul 30 03:11:28.378141 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:28.378108    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:11:28.378267 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:28.378210    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:11:28.378267 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:28.378246    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:11:29.393230 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:29.393189    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p" event={"ID":"8ba45532-1c80-4328-b6d2-c73ff607f539","Type":"ContainerStarted","Data":"e308bca0c77308ff0b0ebbc0b92d78113e69ec80364f3afb8db4f587a49b6ca0"}
Jul 30 03:11:29.560594 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:29.560555    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:29.560804 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:29.560665    2565 patch_prober.go:28] interesting pod/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Jul 30 03:11:29.560804 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:29.560743    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p" podUID="8ba45532-1c80-4328-b6d2-c73ff607f539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Jul 30 03:11:30.560068 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:30.560013    2565 patch_prober.go:28] interesting pod/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Jul 30 03:11:30.560482 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:30.560085    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p" podUID="8ba45532-1c80-4328-b6d2-c73ff607f539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Jul 30 03:11:31.560053 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:31.560008    2565 patch_prober.go:28] interesting pod/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Jul 30 03:11:31.560309 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:31.560100    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p" podUID="8ba45532-1c80-4328-b6d2-c73ff607f539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Jul 30 03:11:32.560815 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:32.560767    2565 patch_prober.go:28] interesting pod/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Jul 30 03:11:32.561263 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:32.560853    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p" podUID="8ba45532-1c80-4328-b6d2-c73ff607f539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Jul 30 03:11:33.563611 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:33.563581    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:33.564071 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:33.564050    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:33.564849 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:33.564830    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:11:35.361324 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:35.361277    2565 patch_prober.go:28] interesting pod/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 30 03:11:35.361811 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:35.361340    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5" podUID="02bb64dc-de3e-4e34-b139-d6510f2eb893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 30 03:11:36.448082 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:36.447978    2565 generic.go:356] "Generic (PLEG): container finished" podID="02bb64dc-de3e-4e34-b139-d6510f2eb893" containerID="1866844b5c2fd1b790ec9cb765f9666484a08bfcc6345763495085fcab2bb331" exitCode=0
Jul 30 03:11:36.448082 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:36.448060    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5" event={"ID":"02bb64dc-de3e-4e34-b139-d6510f2eb893","Type":"ContainerDied","Data":"1866844b5c2fd1b790ec9cb765f9666484a08bfcc6345763495085fcab2bb331"}
Jul 30 03:11:36.448570 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:36.448552    2565 scope.go:117] "RemoveContainer" containerID="1866844b5c2fd1b790ec9cb765f9666484a08bfcc6345763495085fcab2bb331"
Jul 30 03:11:36.449712 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:36.449682    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:11:36.449777 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:36.449753    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:11:36.449814 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:36.449782    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:11:37.454854 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:37.454819    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5" event={"ID":"02bb64dc-de3e-4e34-b139-d6510f2eb893","Type":"ContainerStarted","Data":"30a3a891126007226b019938fd6f6bb1ddb5531bc2a351bc67ff4a850f154072"}
Jul 30 03:11:37.649460 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:37.649424    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:37.649697 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:37.649674    2565 patch_prober.go:28] interesting pod/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Jul 30 03:11:37.649750 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:37.649730    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5" podUID="02bb64dc-de3e-4e34-b139-d6510f2eb893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Jul 30 03:11:38.649774 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:38.649741    2565 patch_prober.go:28] interesting pod/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Jul 30 03:11:38.650198 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:38.649799    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5" podUID="02bb64dc-de3e-4e34-b139-d6510f2eb893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Jul 30 03:11:39.649721 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:39.649680    2565 patch_prober.go:28] interesting pod/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Jul 30 03:11:39.649910 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:39.649744    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5" podUID="02bb64dc-de3e-4e34-b139-d6510f2eb893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Jul 30 03:11:40.649629 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:40.649591    2565 patch_prober.go:28] interesting pod/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Jul 30 03:11:40.649828 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:40.649648    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5" podUID="02bb64dc-de3e-4e34-b139-d6510f2eb893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Jul 30 03:11:41.652411 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:41.652376    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:41.652798 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:41.652715    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:11:41.653523 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:11:41.653502    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:12:05.297421 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:12:05.297390    2565 scope.go:117] "RemoveContainer" containerID="ee6a9cb062b09057ef3f6af069aeb2d7c2065868ce325b6b0f4ee10a84a0b44a"
Jul 30 03:15:00.136986 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:15:00.136948    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f2a805-29756355-tg6qg"]
Jul 30 03:15:00.140687 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:15:00.140669    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f2a805-29756355-tg6qg"
Jul 30 03:15:00.143320 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:15:00.143299    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2npk7\""
Jul 30 03:15:00.150060 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:15:00.150042    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f2a805-29756355-tg6qg"
Jul 30 03:15:00.157821 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:15:00.157795    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f2a805-29756355-tg6qg"]
Jul 30 03:15:00.303113 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:15:00.303081    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9639c0c2_fd3a_4309_9ee1_996060a84ead.slice/crio-3f3093a2e3245aaeac31487e4c56d6d9d16c7646790b72a6fdc8914ecb8e59fe WatchSource:0}: Error finding container 3f3093a2e3245aaeac31487e4c56d6d9d16c7646790b72a6fdc8914ecb8e59fe: Status 404 returned error can't find the container with id 3f3093a2e3245aaeac31487e4c56d6d9d16c7646790b72a6fdc8914ecb8e59fe
Jul 30 03:15:00.303565 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:15:00.303545    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f2a805-29756355-tg6qg"]
Jul 30 03:15:00.634215 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:15:00.634151    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f2a805-29756355-tg6qg" event={"ID":"9639c0c2-fd3a-4309-9ee1-996060a84ead","Type":"ContainerStarted","Data":"3f3093a2e3245aaeac31487e4c56d6d9d16c7646790b72a6fdc8914ecb8e59fe"}
Jul 30 03:15:01.641680 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:15:01.641583    2565 generic.go:356] "Generic (PLEG): container finished" podID="9639c0c2-fd3a-4309-9ee1-996060a84ead" containerID="4d5e779f025c182423c5dc32d071b760448815417c605adbff4c4bf783b4dd9a" exitCode=0
Jul 30 03:15:01.641680 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:15:01.641645    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f2a805-29756355-tg6qg" event={"ID":"9639c0c2-fd3a-4309-9ee1-996060a84ead","Type":"ContainerDied","Data":"4d5e779f025c182423c5dc32d071b760448815417c605adbff4c4bf783b4dd9a"}
Jul 30 03:15:02.783242 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:15:02.783210    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f2a805-29756355-tg6qg"
Jul 30 03:15:03.653954 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:15:03.653929    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f2a805-29756355-tg6qg"
Jul 30 03:15:03.654135 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:15:03.653972    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f2a805-29756355-tg6qg" event={"ID":"9639c0c2-fd3a-4309-9ee1-996060a84ead","Type":"ContainerDied","Data":"3f3093a2e3245aaeac31487e4c56d6d9d16c7646790b72a6fdc8914ecb8e59fe"}
Jul 30 03:15:03.654135 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:15:03.654022    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f3093a2e3245aaeac31487e4c56d6d9d16c7646790b72a6fdc8914ecb8e59fe"
Jul 30 03:20:16.991723 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:16.991634    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-140339311426224"]
Jul 30 03:20:16.992185 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:16.992048    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9639c0c2-fd3a-4309-9ee1-996060a84ead" containerName="cleanup"
Jul 30 03:20:16.992185 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:16.992059    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9639c0c2-fd3a-4309-9ee1-996060a84ead" containerName="cleanup"
Jul 30 03:20:16.992185 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:16.992134    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9639c0c2-fd3a-4309-9ee1-996060a84ead" containerName="cleanup"
Jul 30 03:20:16.995330 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:16.995314    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140339311426224"
Jul 30 03:20:16.998621 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:16.998591    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dkgjx\""
Jul 30 03:20:17.001056 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:17.001032    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-140339311426224"]
Jul 30 03:20:17.118665 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:17.118629    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrjns\" (UniqueName: \"kubernetes.io/projected/e0cd1663-a61f-409d-9e2f-3bad357849fd-kube-api-access-jrjns\") pod \"test-curl-7697-140339311426224\" (UID: \"e0cd1663-a61f-409d-9e2f-3bad357849fd\") " pod="opendatahub/test-curl-7697-140339311426224"
Jul 30 03:20:17.219556 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:17.219514    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrjns\" (UniqueName: \"kubernetes.io/projected/e0cd1663-a61f-409d-9e2f-3bad357849fd-kube-api-access-jrjns\") pod \"test-curl-7697-140339311426224\" (UID: \"e0cd1663-a61f-409d-9e2f-3bad357849fd\") " pod="opendatahub/test-curl-7697-140339311426224"
Jul 30 03:20:17.228510 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:17.228485    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrjns\" (UniqueName: \"kubernetes.io/projected/e0cd1663-a61f-409d-9e2f-3bad357849fd-kube-api-access-jrjns\") pod \"test-curl-7697-140339311426224\" (UID: \"e0cd1663-a61f-409d-9e2f-3bad357849fd\") " pod="opendatahub/test-curl-7697-140339311426224"
Jul 30 03:20:17.307695 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:17.307659    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140339311426224"
Jul 30 03:20:17.452548 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:17.452517    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-140339311426224"]
Jul 30 03:20:17.453846 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:20:17.453803    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0cd1663_a61f_409d_9e2f_3bad357849fd.slice/crio-8c2fc5baa2213280a9ec4756c7928c681c4de716afcf4853ca96072c441057a5 WatchSource:0}: Error finding container 8c2fc5baa2213280a9ec4756c7928c681c4de716afcf4853ca96072c441057a5: Status 404 returned error can't find the container with id 8c2fc5baa2213280a9ec4756c7928c681c4de716afcf4853ca96072c441057a5
Jul 30 03:20:17.455138 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:17.455121    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 03:20:18.462440 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:18.462354    2565 generic.go:356] "Generic (PLEG): container finished" podID="e0cd1663-a61f-409d-9e2f-3bad357849fd" containerID="e9419da25e0707a9fddf27473f5740d9bb89c615471d53d88ea1ffd57d1ca592" exitCode=0
Jul 30 03:20:18.462440 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:18.462405    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-140339311426224" event={"ID":"e0cd1663-a61f-409d-9e2f-3bad357849fd","Type":"ContainerDied","Data":"e9419da25e0707a9fddf27473f5740d9bb89c615471d53d88ea1ffd57d1ca592"}
Jul 30 03:20:18.462834 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:18.462446    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-140339311426224" event={"ID":"e0cd1663-a61f-409d-9e2f-3bad357849fd","Type":"ContainerStarted","Data":"8c2fc5baa2213280a9ec4756c7928c681c4de716afcf4853ca96072c441057a5"}
Jul 30 03:20:19.611499 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:19.611468    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140339311426224"
Jul 30 03:20:19.646537 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:19.646504    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrjns\" (UniqueName: \"kubernetes.io/projected/e0cd1663-a61f-409d-9e2f-3bad357849fd-kube-api-access-jrjns\") pod \"e0cd1663-a61f-409d-9e2f-3bad357849fd\" (UID: \"e0cd1663-a61f-409d-9e2f-3bad357849fd\") "
Jul 30 03:20:19.648767 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:19.648738    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0cd1663-a61f-409d-9e2f-3bad357849fd-kube-api-access-jrjns" (OuterVolumeSpecName: "kube-api-access-jrjns") pod "e0cd1663-a61f-409d-9e2f-3bad357849fd" (UID: "e0cd1663-a61f-409d-9e2f-3bad357849fd"). InnerVolumeSpecName "kube-api-access-jrjns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:20:19.656285 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:19.656233    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7697-140339311426224_e0cd1663-a61f-409d-9e2f-3bad357849fd/test-curl-7697-140339311426224/0.log"
Jul 30 03:20:19.681378 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:19.681338    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-140339311426224"]
Jul 30 03:20:19.683821 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:19.683793    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-140339311426224"]
Jul 30 03:20:19.747531 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:19.747440    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrjns\" (UniqueName: \"kubernetes.io/projected/e0cd1663-a61f-409d-9e2f-3bad357849fd-kube-api-access-jrjns\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:20:19.803081 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:19.803043    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-140339311426352"]
Jul 30 03:20:19.803486 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:19.803472    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0cd1663-a61f-409d-9e2f-3bad357849fd" containerName="test-curl-7697-140339311426224"
Jul 30 03:20:19.803532 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:19.803488    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0cd1663-a61f-409d-9e2f-3bad357849fd" containerName="test-curl-7697-140339311426224"
Jul 30 03:20:19.803567 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:19.803557    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0cd1663-a61f-409d-9e2f-3bad357849fd" containerName="test-curl-7697-140339311426224"
Jul 30 03:20:19.806589 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:19.806573    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140339311426352"
Jul 30 03:20:19.812062 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:19.812034    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-140339311426352"]
Jul 30 03:20:19.848774 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:19.848737    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrmzs\" (UniqueName: \"kubernetes.io/projected/e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb-kube-api-access-nrmzs\") pod \"test-curl-7697-140339311426352\" (UID: \"e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb\") " pod="opendatahub/test-curl-7697-140339311426352"
Jul 30 03:20:19.950161 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:19.950117    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrmzs\" (UniqueName: \"kubernetes.io/projected/e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb-kube-api-access-nrmzs\") pod \"test-curl-7697-140339311426352\" (UID: \"e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb\") " pod="opendatahub/test-curl-7697-140339311426352"
Jul 30 03:20:19.959258 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:19.959234    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrmzs\" (UniqueName: \"kubernetes.io/projected/e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb-kube-api-access-nrmzs\") pod \"test-curl-7697-140339311426352\" (UID: \"e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb\") " pod="opendatahub/test-curl-7697-140339311426352"
Jul 30 03:20:20.117506 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:20.117472    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140339311426352"
Jul 30 03:20:20.261464 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:20.261432    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-140339311426352"]
Jul 30 03:20:20.262708 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:20:20.262686    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode02a3b73_9f4d_4d71_bbe8_ecfc23c76edb.slice/crio-a19906fa57a3bdb46da51e2d8f44cab4e443b5dcc9dd03f075078e572fadc039 WatchSource:0}: Error finding container a19906fa57a3bdb46da51e2d8f44cab4e443b5dcc9dd03f075078e572fadc039: Status 404 returned error can't find the container with id a19906fa57a3bdb46da51e2d8f44cab4e443b5dcc9dd03f075078e572fadc039
Jul 30 03:20:20.475375 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:20.475281    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140339311426224"
Jul 30 03:20:20.475375 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:20.475319    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c2fc5baa2213280a9ec4756c7928c681c4de716afcf4853ca96072c441057a5"
Jul 30 03:20:20.476733 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:20.476699    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-140339311426352" event={"ID":"e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb","Type":"ContainerStarted","Data":"a19906fa57a3bdb46da51e2d8f44cab4e443b5dcc9dd03f075078e572fadc039"}
Jul 30 03:20:20.491629 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:20:20.491599    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7697-140339311426224\" is forbidden: User \"system:node:ip-10-0-132-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-89.ec2.internal' and this object" podUID="e0cd1663-a61f-409d-9e2f-3bad357849fd" pod="opendatahub/test-curl-7697-140339311426224"
Jul 30 03:20:20.801716 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:20.801683    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0cd1663-a61f-409d-9e2f-3bad357849fd" path="/var/lib/kubelet/pods/e0cd1663-a61f-409d-9e2f-3bad357849fd/volumes"
Jul 30 03:20:21.484888 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:21.484855    2565 generic.go:356] "Generic (PLEG): container finished" podID="e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb" containerID="92ba4564f7cf3a6ad5344876e2c7eee2c4f371c7d6f387c6dbe1e4c54829719a" exitCode=0
Jul 30 03:20:21.485097 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:21.484942    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-140339311426352" event={"ID":"e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb","Type":"ContainerDied","Data":"92ba4564f7cf3a6ad5344876e2c7eee2c4f371c7d6f387c6dbe1e4c54829719a"}
Jul 30 03:20:22.627773 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:22.627747    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140339311426352"
Jul 30 03:20:22.647074 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:22.647043    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7697-140339311426352_e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb/test-curl-7697-140339311426352/0.log"
Jul 30 03:20:22.672346 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:22.672314    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-140339311426352"]
Jul 30 03:20:22.675248 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:22.675218    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrmzs\" (UniqueName: \"kubernetes.io/projected/e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb-kube-api-access-nrmzs\") pod \"e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb\" (UID: \"e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb\") "
Jul 30 03:20:22.676349 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:22.676324    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-140339311426352"]
Jul 30 03:20:22.677561 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:22.677540    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb-kube-api-access-nrmzs" (OuterVolumeSpecName: "kube-api-access-nrmzs") pod "e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb" (UID: "e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb"). InnerVolumeSpecName "kube-api-access-nrmzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:20:22.776303 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:22.776218    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrmzs\" (UniqueName: \"kubernetes.io/projected/e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb-kube-api-access-nrmzs\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:20:22.802001 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:22.801962    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb" path="/var/lib/kubelet/pods/e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb/volumes"
Jul 30 03:20:22.836503 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:22.836474    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-140339311366336"]
Jul 30 03:20:22.836878 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:22.836866    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb" containerName="test-curl-7697-140339311426352"
Jul 30 03:20:22.836923 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:22.836880    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb" containerName="test-curl-7697-140339311426352"
Jul 30 03:20:22.836959 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:22.836949    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e02a3b73-9f4d-4d71-bbe8-ecfc23c76edb" containerName="test-curl-7697-140339311426352"
Jul 30 03:20:22.840095 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:22.840079    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140339311366336"
Jul 30 03:20:22.846569 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:22.846547    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-140339311366336"]
Jul 30 03:20:22.877198 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:22.877140    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wwcq\" (UniqueName: \"kubernetes.io/projected/d1567ef3-51c7-427f-988a-414512d36006-kube-api-access-6wwcq\") pod \"test-curl-7697-140339311366336\" (UID: \"d1567ef3-51c7-427f-988a-414512d36006\") " pod="opendatahub/test-curl-7697-140339311366336"
Jul 30 03:20:22.977882 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:22.977850    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wwcq\" (UniqueName: \"kubernetes.io/projected/d1567ef3-51c7-427f-988a-414512d36006-kube-api-access-6wwcq\") pod \"test-curl-7697-140339311366336\" (UID: \"d1567ef3-51c7-427f-988a-414512d36006\") " pod="opendatahub/test-curl-7697-140339311366336"
Jul 30 03:20:22.987318 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:22.987292    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wwcq\" (UniqueName: \"kubernetes.io/projected/d1567ef3-51c7-427f-988a-414512d36006-kube-api-access-6wwcq\") pod \"test-curl-7697-140339311366336\" (UID: \"d1567ef3-51c7-427f-988a-414512d36006\") " pod="opendatahub/test-curl-7697-140339311366336"
Jul 30 03:20:23.150756 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:23.150722    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140339311366336"
Jul 30 03:20:23.289815 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:23.289785    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-140339311366336"]
Jul 30 03:20:23.292267 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:20:23.292240    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1567ef3_51c7_427f_988a_414512d36006.slice/crio-87e929b7aeb2f84fe72240c7e499f4f0b2d48d9154eecd480736df2e9186bb20 WatchSource:0}: Error finding container 87e929b7aeb2f84fe72240c7e499f4f0b2d48d9154eecd480736df2e9186bb20: Status 404 returned error can't find the container with id 87e929b7aeb2f84fe72240c7e499f4f0b2d48d9154eecd480736df2e9186bb20
Jul 30 03:20:23.498662 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:23.498579    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140339311426352"
Jul 30 03:20:23.498662 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:23.498636    2565 scope.go:117] "RemoveContainer" containerID="92ba4564f7cf3a6ad5344876e2c7eee2c4f371c7d6f387c6dbe1e4c54829719a"
Jul 30 03:20:23.500015 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:23.499843    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-140339311366336" event={"ID":"d1567ef3-51c7-427f-988a-414512d36006","Type":"ContainerStarted","Data":"87e929b7aeb2f84fe72240c7e499f4f0b2d48d9154eecd480736df2e9186bb20"}
Jul 30 03:20:24.506919 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:24.506885    2565 generic.go:356] "Generic (PLEG): container finished" podID="d1567ef3-51c7-427f-988a-414512d36006" containerID="4c3345d05c366f78f498e15cacdba0c42a5a8f7e6ea8661fe34140acdc8c2dd5" exitCode=0
Jul 30 03:20:24.507347 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:24.506972    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-140339311366336" event={"ID":"d1567ef3-51c7-427f-988a-414512d36006","Type":"ContainerDied","Data":"4c3345d05c366f78f498e15cacdba0c42a5a8f7e6ea8661fe34140acdc8c2dd5"}
Jul 30 03:20:25.660597 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:25.660571    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140339311366336"
Jul 30 03:20:25.682295 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:25.682237    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7697-140339311366336_d1567ef3-51c7-427f-988a-414512d36006/test-curl-7697-140339311366336/0.log"
Jul 30 03:20:25.703562 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:25.703523    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wwcq\" (UniqueName: \"kubernetes.io/projected/d1567ef3-51c7-427f-988a-414512d36006-kube-api-access-6wwcq\") pod \"d1567ef3-51c7-427f-988a-414512d36006\" (UID: \"d1567ef3-51c7-427f-988a-414512d36006\") "
Jul 30 03:20:25.706541 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:25.706512    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1567ef3-51c7-427f-988a-414512d36006-kube-api-access-6wwcq" (OuterVolumeSpecName: "kube-api-access-6wwcq") pod "d1567ef3-51c7-427f-988a-414512d36006" (UID: "d1567ef3-51c7-427f-988a-414512d36006"). InnerVolumeSpecName "kube-api-access-6wwcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:20:25.708132 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:25.708109    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-140339311366336"]
Jul 30 03:20:25.711873 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:25.711847    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-140339311366336"]
Jul 30 03:20:25.804979 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:25.804938    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wwcq\" (UniqueName: \"kubernetes.io/projected/d1567ef3-51c7-427f-988a-414512d36006-kube-api-access-6wwcq\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:20:25.829288 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:25.829258    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-140339311366336"]
Jul 30 03:20:25.829693 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:25.829680    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1567ef3-51c7-427f-988a-414512d36006" containerName="test-curl-7697-140339311366336"
Jul 30 03:20:25.829742 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:25.829695    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1567ef3-51c7-427f-988a-414512d36006" containerName="test-curl-7697-140339311366336"
Jul 30 03:20:25.829777 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:25.829768    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1567ef3-51c7-427f-988a-414512d36006" containerName="test-curl-7697-140339311366336"
Jul 30 03:20:25.832867 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:25.832850    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140339311366336"
Jul 30 03:20:25.837133 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:25.837105    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-140339311366336"]
Jul 30 03:20:25.905931 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:25.905888    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmq8t\" (UniqueName: \"kubernetes.io/projected/4445a414-9ef1-47a7-a427-03ddb7576158-kube-api-access-nmq8t\") pod \"test-curl-7697-140339311366336\" (UID: \"4445a414-9ef1-47a7-a427-03ddb7576158\") " pod="opendatahub/test-curl-7697-140339311366336"
Jul 30 03:20:26.007228 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:26.007192    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmq8t\" (UniqueName: \"kubernetes.io/projected/4445a414-9ef1-47a7-a427-03ddb7576158-kube-api-access-nmq8t\") pod \"test-curl-7697-140339311366336\" (UID: \"4445a414-9ef1-47a7-a427-03ddb7576158\") " pod="opendatahub/test-curl-7697-140339311366336"
Jul 30 03:20:26.016096 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:26.016064    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmq8t\" (UniqueName: \"kubernetes.io/projected/4445a414-9ef1-47a7-a427-03ddb7576158-kube-api-access-nmq8t\") pod \"test-curl-7697-140339311366336\" (UID: \"4445a414-9ef1-47a7-a427-03ddb7576158\") " pod="opendatahub/test-curl-7697-140339311366336"
Jul 30 03:20:26.144077 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:26.143991    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140339311366336"
Jul 30 03:20:26.279533 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:26.279505    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-140339311366336"]
Jul 30 03:20:26.280846 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:20:26.280817    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4445a414_9ef1_47a7_a427_03ddb7576158.slice/crio-efe19a6ff5d875ab6e2eeaa891b210ab124bf51e23eaf3a5a36fa65621b03eb7 WatchSource:0}: Error finding container efe19a6ff5d875ab6e2eeaa891b210ab124bf51e23eaf3a5a36fa65621b03eb7: Status 404 returned error can't find the container with id efe19a6ff5d875ab6e2eeaa891b210ab124bf51e23eaf3a5a36fa65621b03eb7
Jul 30 03:20:26.520289 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:26.520210    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140339311366336"
Jul 30 03:20:26.520459 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:26.520235    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87e929b7aeb2f84fe72240c7e499f4f0b2d48d9154eecd480736df2e9186bb20"
Jul 30 03:20:26.521562 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:26.521534    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-140339311366336" event={"ID":"4445a414-9ef1-47a7-a427-03ddb7576158","Type":"ContainerStarted","Data":"efe19a6ff5d875ab6e2eeaa891b210ab124bf51e23eaf3a5a36fa65621b03eb7"}
Jul 30 03:20:26.532024 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:26.531993    2565 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7697-140339311366336" oldPodUID="d1567ef3-51c7-427f-988a-414512d36006" podUID="4445a414-9ef1-47a7-a427-03ddb7576158"
Jul 30 03:20:26.801231 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:26.801196    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1567ef3-51c7-427f-988a-414512d36006" path="/var/lib/kubelet/pods/d1567ef3-51c7-427f-988a-414512d36006/volumes"
Jul 30 03:20:27.529569 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:27.529534    2565 generic.go:356] "Generic (PLEG): container finished" podID="4445a414-9ef1-47a7-a427-03ddb7576158" containerID="c444ffe5c0f391a2a11e154bc73d29374d8e9e7f4268e715b344c9c26b1f0add" exitCode=0
Jul 30 03:20:27.529747 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:27.529621    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-140339311366336" event={"ID":"4445a414-9ef1-47a7-a427-03ddb7576158","Type":"ContainerDied","Data":"c444ffe5c0f391a2a11e154bc73d29374d8e9e7f4268e715b344c9c26b1f0add"}
Jul 30 03:20:28.676495 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:28.676468    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140339311366336"
Jul 30 03:20:28.697286 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:28.697254    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7697-140339311366336_4445a414-9ef1-47a7-a427-03ddb7576158/test-curl-7697-140339311366336/0.log"
Jul 30 03:20:28.719775 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:28.719698    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-140339311366336"]
Jul 30 03:20:28.722633 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:28.722606    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-140339311366336"]
Jul 30 03:20:28.736193 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:28.736150    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmq8t\" (UniqueName: \"kubernetes.io/projected/4445a414-9ef1-47a7-a427-03ddb7576158-kube-api-access-nmq8t\") pod \"4445a414-9ef1-47a7-a427-03ddb7576158\" (UID: \"4445a414-9ef1-47a7-a427-03ddb7576158\") "
Jul 30 03:20:28.738349 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:28.738315    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4445a414-9ef1-47a7-a427-03ddb7576158-kube-api-access-nmq8t" (OuterVolumeSpecName: "kube-api-access-nmq8t") pod "4445a414-9ef1-47a7-a427-03ddb7576158" (UID: "4445a414-9ef1-47a7-a427-03ddb7576158"). InnerVolumeSpecName "kube-api-access-nmq8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:20:28.803097 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:28.803056    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4445a414-9ef1-47a7-a427-03ddb7576158" path="/var/lib/kubelet/pods/4445a414-9ef1-47a7-a427-03ddb7576158/volumes"
Jul 30 03:20:28.837242 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:28.837202    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmq8t\" (UniqueName: \"kubernetes.io/projected/4445a414-9ef1-47a7-a427-03ddb7576158-kube-api-access-nmq8t\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:20:28.839593 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:28.839572    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-140339311367344"]
Jul 30 03:20:28.839979 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:28.839965    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4445a414-9ef1-47a7-a427-03ddb7576158" containerName="test-curl-7697-140339311366336"
Jul 30 03:20:28.840038 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:28.839981    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4445a414-9ef1-47a7-a427-03ddb7576158" containerName="test-curl-7697-140339311366336"
Jul 30 03:20:28.840079 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:28.840048    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4445a414-9ef1-47a7-a427-03ddb7576158" containerName="test-curl-7697-140339311366336"
Jul 30 03:20:28.843293 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:28.843276    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140339311367344"
Jul 30 03:20:28.849395 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:28.849374    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-140339311367344"]
Jul 30 03:20:28.938459 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:28.938419    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8cjb\" (UniqueName: \"kubernetes.io/projected/7e78385b-a9a2-425e-8937-73734cfcb238-kube-api-access-s8cjb\") pod \"test-curl-7697-140339311367344\" (UID: \"7e78385b-a9a2-425e-8937-73734cfcb238\") " pod="opendatahub/test-curl-7697-140339311367344"
Jul 30 03:20:29.039412 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:29.039370    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8cjb\" (UniqueName: \"kubernetes.io/projected/7e78385b-a9a2-425e-8937-73734cfcb238-kube-api-access-s8cjb\") pod \"test-curl-7697-140339311367344\" (UID: \"7e78385b-a9a2-425e-8937-73734cfcb238\") " pod="opendatahub/test-curl-7697-140339311367344"
Jul 30 03:20:29.047205 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:29.047153    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8cjb\" (UniqueName: \"kubernetes.io/projected/7e78385b-a9a2-425e-8937-73734cfcb238-kube-api-access-s8cjb\") pod \"test-curl-7697-140339311367344\" (UID: \"7e78385b-a9a2-425e-8937-73734cfcb238\") " pod="opendatahub/test-curl-7697-140339311367344"
Jul 30 03:20:29.154233 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:29.154146    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140339311367344"
Jul 30 03:20:29.293898 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:29.293866    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-140339311367344"]
Jul 30 03:20:29.295362 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:20:29.295332    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e78385b_a9a2_425e_8937_73734cfcb238.slice/crio-b6f25c585a47d03b671f13ac661adb103d4a27f559ac8ea9b379ee714623fd5a WatchSource:0}: Error finding container b6f25c585a47d03b671f13ac661adb103d4a27f559ac8ea9b379ee714623fd5a: Status 404 returned error can't find the container with id b6f25c585a47d03b671f13ac661adb103d4a27f559ac8ea9b379ee714623fd5a
Jul 30 03:20:29.542748 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:29.542714    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-140339311367344" event={"ID":"7e78385b-a9a2-425e-8937-73734cfcb238","Type":"ContainerStarted","Data":"b6f25c585a47d03b671f13ac661adb103d4a27f559ac8ea9b379ee714623fd5a"}
Jul 30 03:20:29.543888 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:29.543867    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140339311366336"
Jul 30 03:20:29.543888 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:29.543863    2565 scope.go:117] "RemoveContainer" containerID="c444ffe5c0f391a2a11e154bc73d29374d8e9e7f4268e715b344c9c26b1f0add"
Jul 30 03:20:30.554013 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:30.553974    2565 generic.go:356] "Generic (PLEG): container finished" podID="7e78385b-a9a2-425e-8937-73734cfcb238" containerID="af8d87346f178b1974b67ce30477334f2b581b076b715cea3919bb3223f92f31" exitCode=0
Jul 30 03:20:30.554472 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:30.554048    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-140339311367344" event={"ID":"7e78385b-a9a2-425e-8937-73734cfcb238","Type":"ContainerDied","Data":"af8d87346f178b1974b67ce30477334f2b581b076b715cea3919bb3223f92f31"}
Jul 30 03:20:31.698765 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:31.698740    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140339311367344"
Jul 30 03:20:31.718987 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:31.718956    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7697-140339311367344_7e78385b-a9a2-425e-8937-73734cfcb238/test-curl-7697-140339311367344/0.log"
Jul 30 03:20:31.742679 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:31.742631    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-140339311367344"]
Jul 30 03:20:31.745185 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:31.745146    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-140339311367344"]
Jul 30 03:20:31.766529 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:31.766497    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8cjb\" (UniqueName: \"kubernetes.io/projected/7e78385b-a9a2-425e-8937-73734cfcb238-kube-api-access-s8cjb\") pod \"7e78385b-a9a2-425e-8937-73734cfcb238\" (UID: \"7e78385b-a9a2-425e-8937-73734cfcb238\") "
Jul 30 03:20:31.768730 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:31.768695    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e78385b-a9a2-425e-8937-73734cfcb238-kube-api-access-s8cjb" (OuterVolumeSpecName: "kube-api-access-s8cjb") pod "7e78385b-a9a2-425e-8937-73734cfcb238" (UID: "7e78385b-a9a2-425e-8937-73734cfcb238"). InnerVolumeSpecName "kube-api-access-s8cjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:20:31.867495 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:31.867407    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8cjb\" (UniqueName: \"kubernetes.io/projected/7e78385b-a9a2-425e-8937-73734cfcb238-kube-api-access-s8cjb\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:20:32.567303 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:32.567277    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-140339311367344"
Jul 30 03:20:32.567303 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:32.567294    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6f25c585a47d03b671f13ac661adb103d4a27f559ac8ea9b379ee714623fd5a"
Jul 30 03:20:32.577354 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:20:32.577324    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7697-140339311367344\" is forbidden: User \"system:node:ip-10-0-132-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-89.ec2.internal' and this object" podUID="7e78385b-a9a2-425e-8937-73734cfcb238" pod="opendatahub/test-curl-7697-140339311367344"
Jul 30 03:20:32.801315 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:32.801282    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e78385b-a9a2-425e-8937-73734cfcb238" path="/var/lib/kubelet/pods/7e78385b-a9a2-425e-8937-73734cfcb238/volumes"
Jul 30 03:20:35.270951 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.270917    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"]
Jul 30 03:20:35.271436 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.271415    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e78385b-a9a2-425e-8937-73734cfcb238" containerName="test-curl-7697-140339311367344"
Jul 30 03:20:35.271504 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.271440    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e78385b-a9a2-425e-8937-73734cfcb238" containerName="test-curl-7697-140339311367344"
Jul 30 03:20:35.271569 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.271557    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e78385b-a9a2-425e-8937-73734cfcb238" containerName="test-curl-7697-140339311367344"
Jul 30 03:20:35.275315 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.275290    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.282205 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.278392    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2cad57-openshift-default-dockercfg-gfrnn\""
Jul 30 03:20:35.291886 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.291856    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"]
Jul 30 03:20:35.400738 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.400624    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-token\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.400964 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.400782    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-podinfo\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.400964 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.400819    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-workload-certs\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.400964 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.400846    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-envoy\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.400964 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.400884    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-credential-socket\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.400964 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.400955    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-workload-socket\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.401288 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.401005    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7b8f82-f331-4697-be50-4f724fd68bd0-istiod-ca-cert\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.401288 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.401036    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-data\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.401288 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.401073    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwx87\" (UniqueName: \"kubernetes.io/projected/8d7b8f82-f331-4697-be50-4f724fd68bd0-kube-api-access-gwx87\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.501805 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.501767    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-token\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.502017 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.501823    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-podinfo\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.502017 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.501853    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-workload-certs\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.502017 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.501878    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-envoy\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.502017 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.501912    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-credential-socket\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.502017 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.501968    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-workload-socket\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.502017 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.502011    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7b8f82-f331-4697-be50-4f724fd68bd0-istiod-ca-cert\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.502407 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.502042    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-data\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.502407 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.502073    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwx87\" (UniqueName: \"kubernetes.io/projected/8d7b8f82-f331-4697-be50-4f724fd68bd0-kube-api-access-gwx87\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.502514 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.502412    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-workload-certs\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.502514 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.502437    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-credential-socket\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.502514 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.502480    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-workload-socket\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.502809 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.502790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7b8f82-f331-4697-be50-4f724fd68bd0-istiod-ca-cert\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.502852 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.502756    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-data\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.504385 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.504365    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-envoy\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.504699 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.504680    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-podinfo\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.509639 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.509617    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwx87\" (UniqueName: \"kubernetes.io/projected/8d7b8f82-f331-4697-be50-4f724fd68bd0-kube-api-access-gwx87\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.509731 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.509616    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-token\") pod \"e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") " pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.598061 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.598024    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:35.748887 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.748862    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"]
Jul 30 03:20:35.750995 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:20:35.750967    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7b8f82_f331_4697_be50_4f724fd68bd0.slice/crio-5600c60e45fbf829c5146ef966382b716518c5ab0306c2a42ae2d51b8da0bc8a WatchSource:0}: Error finding container 5600c60e45fbf829c5146ef966382b716518c5ab0306c2a42ae2d51b8da0bc8a: Status 404 returned error can't find the container with id 5600c60e45fbf829c5146ef966382b716518c5ab0306c2a42ae2d51b8da0bc8a
Jul 30 03:20:35.754075 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.754036    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:20:35.754190 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.754136    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:20:35.754265 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:35.754245    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:20:36.592607 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:36.592568    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29" event={"ID":"8d7b8f82-f331-4697-be50-4f724fd68bd0","Type":"ContainerStarted","Data":"18b67542ab75290543d3c786c7f3e8e208feaab026c11d1078d847de16b3327a"}
Jul 30 03:20:36.593008 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:36.592625    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29" event={"ID":"8d7b8f82-f331-4697-be50-4f724fd68bd0","Type":"ContainerStarted","Data":"5600c60e45fbf829c5146ef966382b716518c5ab0306c2a42ae2d51b8da0bc8a"}
Jul 30 03:20:36.599222 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:36.599200    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:36.604514 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:36.604490    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:36.616108 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:36.616052    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29" podStartSLOduration=1.616039419 podStartE2EDuration="1.616039419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:20:36.613053995 +0000 UTC m=+3272.259837424" watchObservedRunningTime="2026-07-30 03:20:36.616039419 +0000 UTC m=+3272.262822823"
Jul 30 03:20:37.598189 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:37.598144    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:37.599329 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:37.599300    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:38.604141 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:38.604110    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"]
Jul 30 03:20:38.608495 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:38.608470    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"
Jul 30 03:20:38.610985 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:38.610961    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2cad57-serving-cert\""
Jul 30 03:20:38.615938 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:38.615914    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"]
Jul 30 03:20:38.732136 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:38.732096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvlrr\" (UniqueName: \"kubernetes.io/projected/9775762d-34b7-4161-986a-d462a991aff0-kube-api-access-hvlrr\") pod \"maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d\" (UID: \"9775762d-34b7-4161-986a-d462a991aff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"
Jul 30 03:20:38.732136 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:38.732135    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9775762d-34b7-4161-986a-d462a991aff0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d\" (UID: \"9775762d-34b7-4161-986a-d462a991aff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"
Jul 30 03:20:38.732400 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:38.732330    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9775762d-34b7-4161-986a-d462a991aff0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d\" (UID: \"9775762d-34b7-4161-986a-d462a991aff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"
Jul 30 03:20:38.833795 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:38.833755    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvlrr\" (UniqueName: \"kubernetes.io/projected/9775762d-34b7-4161-986a-d462a991aff0-kube-api-access-hvlrr\") pod \"maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d\" (UID: \"9775762d-34b7-4161-986a-d462a991aff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"
Jul 30 03:20:38.833795 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:38.833796    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9775762d-34b7-4161-986a-d462a991aff0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d\" (UID: \"9775762d-34b7-4161-986a-d462a991aff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"
Jul 30 03:20:38.834007 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:38.833876    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9775762d-34b7-4161-986a-d462a991aff0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d\" (UID: \"9775762d-34b7-4161-986a-d462a991aff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"
Jul 30 03:20:38.834053 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:20:38.834032    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-serving-cert: secret "maas-api-e2e-ipp-a-2cad57-serving-cert" not found
Jul 30 03:20:38.834124 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:20:38.834103    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9775762d-34b7-4161-986a-d462a991aff0-maas-api-tls podName:9775762d-34b7-4161-986a-d462a991aff0 nodeName:}" failed. No retries permitted until 2026-07-30 03:20:39.334084887 +0000 UTC m=+3274.980868269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9775762d-34b7-4161-986a-d462a991aff0-maas-api-tls") pod "maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d" (UID: "9775762d-34b7-4161-986a-d462a991aff0") : secret "maas-api-e2e-ipp-a-2cad57-serving-cert" not found
Jul 30 03:20:38.834621 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:38.834603    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9775762d-34b7-4161-986a-d462a991aff0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d\" (UID: \"9775762d-34b7-4161-986a-d462a991aff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"
Jul 30 03:20:38.845440 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:38.845415    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvlrr\" (UniqueName: \"kubernetes.io/projected/9775762d-34b7-4161-986a-d462a991aff0-kube-api-access-hvlrr\") pod \"maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d\" (UID: \"9775762d-34b7-4161-986a-d462a991aff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"
Jul 30 03:20:39.339234 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:39.339160    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9775762d-34b7-4161-986a-d462a991aff0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d\" (UID: \"9775762d-34b7-4161-986a-d462a991aff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"
Jul 30 03:20:39.341821 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:39.341794    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9775762d-34b7-4161-986a-d462a991aff0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d\" (UID: \"9775762d-34b7-4161-986a-d462a991aff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"
Jul 30 03:20:39.521993 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:39.521964    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"
Jul 30 03:20:39.678648 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:39.678610    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"]
Jul 30 03:20:39.679442 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:20:39.679402    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9775762d_34b7_4161_986a_d462a991aff0.slice/crio-3579cd8c77b1bf5fccb7fd3a442c01c1d338b7e9bd2e1b94e01a676eb07fd716 WatchSource:0}: Error finding container 3579cd8c77b1bf5fccb7fd3a442c01c1d338b7e9bd2e1b94e01a676eb07fd716: Status 404 returned error can't find the container with id 3579cd8c77b1bf5fccb7fd3a442c01c1d338b7e9bd2e1b94e01a676eb07fd716
Jul 30 03:20:40.619309 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:40.619275    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d" event={"ID":"9775762d-34b7-4161-986a-d462a991aff0","Type":"ContainerStarted","Data":"e1376c44a8106e4033b107f0606c31cfe598c47afb34a21b72226424e6da8988"}
Jul 30 03:20:40.619491 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:40.619359    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d" event={"ID":"9775762d-34b7-4161-986a-d462a991aff0","Type":"ContainerStarted","Data":"3579cd8c77b1bf5fccb7fd3a442c01c1d338b7e9bd2e1b94e01a676eb07fd716"}
Jul 30 03:20:40.619491 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:40.619395    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"
Jul 30 03:20:40.637678 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:40.637614    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d" podStartSLOduration=2.156593161 podStartE2EDuration="2.637598144s" totalImagesPullingTime="481.004983ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:20:40.63571464 +0000 UTC m=+3276.282498045" watchObservedRunningTime="2026-07-30 03:20:40.637598144 +0000 UTC m=+3276.284381558"
Jul 30 03:20:46.631750 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:46.631718    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"
Jul 30 03:20:53.639392 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:53.639347    2565 patch_prober.go:28] interesting pod/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 30 03:20:53.639855 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:53.639436    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29" podUID="8d7b8f82-f331-4697-be50-4f724fd68bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 30 03:20:54.241462 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.241430    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"]
Jul 30 03:20:54.246726 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.246706    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.249880 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.249855    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f58ab6-openshift-default-dockercfg-7h2bs\""
Jul 30 03:20:54.260874 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.260849    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"]
Jul 30 03:20:54.388927 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.388888    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-workload-certs\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.389141 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.388932    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-token\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.389141 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.389018    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3b9928-470e-40d0-a120-8041725bf0e3-istiod-ca-cert\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.389141 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.389082    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-workload-socket\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.389141 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.389125    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-podinfo\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.389376 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.389155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-envoy\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.389376 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.389240    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-data\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.389376 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.389316    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-credential-socket\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.389376 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.389363    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7qn4\" (UniqueName: \"kubernetes.io/projected/7c3b9928-470e-40d0-a120-8041725bf0e3-kube-api-access-j7qn4\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.490041 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.490000    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7qn4\" (UniqueName: \"kubernetes.io/projected/7c3b9928-470e-40d0-a120-8041725bf0e3-kube-api-access-j7qn4\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.490277 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.490055    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-workload-certs\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.490277 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.490077    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-token\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.490277 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.490122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3b9928-470e-40d0-a120-8041725bf0e3-istiod-ca-cert\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.490277 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.490155    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-workload-socket\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.490277 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.490204    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-podinfo\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.490277 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.490237    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-envoy\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.490277 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.490262    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-data\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.490662 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.490354    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-credential-socket\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.490662 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.490629    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-workload-certs\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.490771 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.490707    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-data\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.490771 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.490752    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-credential-socket\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.490977 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.490953    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-workload-socket\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.491236 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.491158    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3b9928-470e-40d0-a120-8041725bf0e3-istiod-ca-cert\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.492908 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.492844    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-envoy\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.493019 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.492958    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-podinfo\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.498925 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.498902    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7qn4\" (UniqueName: \"kubernetes.io/projected/7c3b9928-470e-40d0-a120-8041725bf0e3-kube-api-access-j7qn4\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.499028 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.498956    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-token\") pod \"e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") " pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.558541 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.558496    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:54.716371 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.716344    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"]
Jul 30 03:20:54.717643 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:20:54.717615    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3b9928_470e_40d0_a120_8041725bf0e3.slice/crio-c6e63bf780690b31c5e469337ee711f8b02b1467b70399f9db4d57d60677e0c6 WatchSource:0}: Error finding container c6e63bf780690b31c5e469337ee711f8b02b1467b70399f9db4d57d60677e0c6: Status 404 returned error can't find the container with id c6e63bf780690b31c5e469337ee711f8b02b1467b70399f9db4d57d60677e0c6
Jul 30 03:20:54.720001 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.719968    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:20:54.720108 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.720033    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:20:54.720108 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:54.720062    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:20:55.722301 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:55.722268    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d" event={"ID":"7c3b9928-470e-40d0-a120-8041725bf0e3","Type":"ContainerStarted","Data":"3b5d1e6142b7c5ebbe94383cfa317d3a14f1b4d6f0e5b6276f567856f71a8355"}
Jul 30 03:20:55.722816 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:55.722319    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d" event={"ID":"7c3b9928-470e-40d0-a120-8041725bf0e3","Type":"ContainerStarted","Data":"c6e63bf780690b31c5e469337ee711f8b02b1467b70399f9db4d57d60677e0c6"}
Jul 30 03:20:55.723880 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:55.723856    2565 generic.go:356] "Generic (PLEG): container finished" podID="8d7b8f82-f331-4697-be50-4f724fd68bd0" containerID="18b67542ab75290543d3c786c7f3e8e208feaab026c11d1078d847de16b3327a" exitCode=0
Jul 30 03:20:55.724004 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:55.723881    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29" event={"ID":"8d7b8f82-f331-4697-be50-4f724fd68bd0","Type":"ContainerDied","Data":"18b67542ab75290543d3c786c7f3e8e208feaab026c11d1078d847de16b3327a"}
Jul 30 03:20:55.724296 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:55.724280    2565 scope.go:117] "RemoveContainer" containerID="18b67542ab75290543d3c786c7f3e8e208feaab026c11d1078d847de16b3327a"
Jul 30 03:20:55.725391 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:55.725359    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:20:55.725647 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:55.725445    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:20:55.725647 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:55.725499    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:20:55.757655 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:55.757559    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d" podStartSLOduration=1.757543243 podStartE2EDuration="1.757543243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:20:55.754912867 +0000 UTC m=+3291.401696297" watchObservedRunningTime="2026-07-30 03:20:55.757543243 +0000 UTC m=+3291.404326648"
Jul 30 03:20:56.559390 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:56.559353    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:56.564369 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:56.564343    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:56.731987 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:56.731950    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29" event={"ID":"8d7b8f82-f331-4697-be50-4f724fd68bd0","Type":"ContainerStarted","Data":"7072c802e1f25b5be38b3ee4fa7a5ded31f8adc8f8052bab80bd8c341b6656e1"}
Jul 30 03:20:56.732501 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:56.732378    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:56.733186 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:56.733147    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:20:57.599042 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:57.598990    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:20:57.599270 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:57.599250    2565 patch_prober.go:28] interesting pod/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Jul 30 03:20:57.599333 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:57.599297    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29" podUID="8d7b8f82-f331-4697-be50-4f724fd68bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Jul 30 03:20:58.599139 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:58.599093    2565 patch_prober.go:28] interesting pod/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Jul 30 03:20:58.599663 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:58.599571    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29" podUID="8d7b8f82-f331-4697-be50-4f724fd68bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Jul 30 03:20:59.599057 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:59.599012    2565 patch_prober.go:28] interesting pod/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Jul 30 03:20:59.599310 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:20:59.599084    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29" podUID="8d7b8f82-f331-4697-be50-4f724fd68bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Jul 30 03:21:00.598913 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:00.598871    2565 patch_prober.go:28] interesting pod/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Jul 30 03:21:00.599097 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:00.598943    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29" podUID="8d7b8f82-f331-4697-be50-4f724fd68bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Jul 30 03:21:01.602334 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:01.602300    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:21:01.602740 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:01.602704    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:21:01.603371 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:01.603354    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:21:10.841511 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:10.841477    2565 generic.go:356] "Generic (PLEG): container finished" podID="7c3b9928-470e-40d0-a120-8041725bf0e3" containerID="3b5d1e6142b7c5ebbe94383cfa317d3a14f1b4d6f0e5b6276f567856f71a8355" exitCode=0
Jul 30 03:21:10.841942 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:10.841562    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d" event={"ID":"7c3b9928-470e-40d0-a120-8041725bf0e3","Type":"ContainerDied","Data":"3b5d1e6142b7c5ebbe94383cfa317d3a14f1b4d6f0e5b6276f567856f71a8355"}
Jul 30 03:21:10.842021 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:10.841991    2565 scope.go:117] "RemoveContainer" containerID="3b5d1e6142b7c5ebbe94383cfa317d3a14f1b4d6f0e5b6276f567856f71a8355"
Jul 30 03:21:10.843104 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:10.843077    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:21:10.843183 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:10.843149    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:21:10.843230 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:10.843210    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:21:11.733078 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:11.733044    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:21:11.849619 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:11.849582    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d" event={"ID":"7c3b9928-470e-40d0-a120-8041725bf0e3","Type":"ContainerStarted","Data":"9d81bbf20112044fcd51049affaa012282cb7757955a990ddca960ec441baa2f"}
Jul 30 03:21:12.559487 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:12.559450    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:21:12.560673 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:12.560648    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Jul 30 03:21:12.560787 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:12.560697    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d" podUID="7c3b9928-470e-40d0-a120-8041725bf0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Jul 30 03:21:13.559131 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:13.559089    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Jul 30 03:21:13.559577 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:13.559157    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d" podUID="7c3b9928-470e-40d0-a120-8041725bf0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Jul 30 03:21:14.559870 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:14.559833    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Jul 30 03:21:14.560318 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:14.559892    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d" podUID="7c3b9928-470e-40d0-a120-8041725bf0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Jul 30 03:21:15.559439 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:15.559401    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Jul 30 03:21:15.559634 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:15.559458    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d" podUID="7c3b9928-470e-40d0-a120-8041725bf0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Jul 30 03:21:16.563181 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:16.563137    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:21:16.563584 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:16.563425    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:21:16.564117 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:16.564101    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:21:29.478972 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:21:29.478891    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-wg4rd_0c013301-ce62-4de8-87b6-4e9e6ed43dba/payload-processing/0.log"
Jul 30 03:22:24.659240 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:24.659201    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-wg4rd_0c013301-ce62-4de8-87b6-4e9e6ed43dba/payload-processing/0.log"
Jul 30 03:22:25.392698 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.392660    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"]
Jul 30 03:22:25.395274 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.395255    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.399534 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.399510    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-64f2f8-openshift-default-dockercfg-vfd48\""
Jul 30 03:22:25.421075 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.421037    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"]
Jul 30 03:22:25.527410 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.527373    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-envoy\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.527410 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.527415    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkkfp\" (UniqueName: \"kubernetes.io/projected/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-kube-api-access-mkkfp\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.527625 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.527451    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.527625 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.527492    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-workload-socket\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.527625 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.527521    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-data\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.527625 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.527582    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-podinfo\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.527625 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.527608    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-workload-certs\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.527789 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.527633    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-credential-socket\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.527789 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.527690    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-token\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.628875 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.628843    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-podinfo\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.629083 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.628892    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-workload-certs\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.629083 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.629045    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-credential-socket\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.629252 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.629128    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-token\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.629312 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.629255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-envoy\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.629312 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.629293    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkkfp\" (UniqueName: \"kubernetes.io/projected/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-kube-api-access-mkkfp\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.629415 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.629261    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-workload-certs\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.629415 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.629344    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.629415 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.629360    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-credential-socket\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.629415 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.629396    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-workload-socket\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.629628 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.629432    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-data\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.629730 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.629704    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-workload-socket\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.629846 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.629819    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-data\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.630004 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.629976    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.632066 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.632042    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-envoy\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.632297 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.632282    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-podinfo\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.638533 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.638505    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-token\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.639220 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.639200    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkkfp\" (UniqueName: \"kubernetes.io/projected/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-kube-api-access-mkkfp\") pod \"e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") " pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.705870 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.705777    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:25.869571 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.869534    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"]
Jul 30 03:22:25.869930 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:22:25.869909    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74a1dac_06b5_4ca7_b2a3_a624549d6f1d.slice/crio-b836d5e4d9234c131b9ccd45301276f846bb0b3d04b8b49b988444fefd8f2128 WatchSource:0}: Error finding container b836d5e4d9234c131b9ccd45301276f846bb0b3d04b8b49b988444fefd8f2128: Status 404 returned error can't find the container with id b836d5e4d9234c131b9ccd45301276f846bb0b3d04b8b49b988444fefd8f2128
Jul 30 03:22:25.872509 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.872471    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:22:25.872594 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.872557    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:22:25.872637 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:25.872601    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:22:26.328751 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:26.328710    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg" event={"ID":"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d","Type":"ContainerStarted","Data":"1065cde4c485adb6f6a7dfea56cfaab4fb2f5b73ce4b81544d7cee6b6f43f5db"}
Jul 30 03:22:26.328979 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:26.328810    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg" event={"ID":"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d","Type":"ContainerStarted","Data":"b836d5e4d9234c131b9ccd45301276f846bb0b3d04b8b49b988444fefd8f2128"}
Jul 30 03:22:26.349364 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:26.349313    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg" podStartSLOduration=1.349295607 podStartE2EDuration="1.349295607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:22:26.34751817 +0000 UTC m=+3381.994301616" watchObservedRunningTime="2026-07-30 03:22:26.349295607 +0000 UTC m=+3381.996079012"
Jul 30 03:22:26.706652 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:26.706549    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:26.711510 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:26.711484    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:27.335656 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:27.335625    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:27.336766 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:27.336747    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:42.337816 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:42.337723    2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Jul 30 03:22:42.338259 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:42.337814    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg" podUID="a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Jul 30 03:22:42.438816 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:42.438784    2565 generic.go:356] "Generic (PLEG): container finished" podID="a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" containerID="1065cde4c485adb6f6a7dfea56cfaab4fb2f5b73ce4b81544d7cee6b6f43f5db" exitCode=0
Jul 30 03:22:42.439018 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:42.438862    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg" event={"ID":"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d","Type":"ContainerDied","Data":"1065cde4c485adb6f6a7dfea56cfaab4fb2f5b73ce4b81544d7cee6b6f43f5db"}
Jul 30 03:22:42.439392 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:42.439375    2565 scope.go:117] "RemoveContainer" containerID="1065cde4c485adb6f6a7dfea56cfaab4fb2f5b73ce4b81544d7cee6b6f43f5db"
Jul 30 03:22:42.440574 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:42.440549    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:22:42.440634 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:42.440617    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:22:42.440670 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:42.440647    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 30 03:22:43.448002 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:43.447970    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg" event={"ID":"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d","Type":"ContainerStarted","Data":"feba2c2c6a4375c3d5cda2d2d41e105b9547716e966b1abddfea8afbb6c0d782"}
Jul 30 03:22:43.707121 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:43.706630    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:43.707121 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:43.706761    2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Jul 30 03:22:43.707121 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:43.706804    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg" podUID="a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Jul 30 03:22:44.706278 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:44.706238    2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Jul 30 03:22:44.706708 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:44.706333    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg" podUID="a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Jul 30 03:22:45.706046 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:45.706013    2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Jul 30 03:22:45.706248 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:45.706067    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg" podUID="a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Jul 30 03:22:46.707107 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:46.707073    2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Jul 30 03:22:46.707538 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:46.707146    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg" podUID="a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Jul 30 03:22:47.709909 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:47.709877    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:47.710358 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:47.710153    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:22:47.710810 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:22:47.710790    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:23:05.133859 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:05.133758    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"]
Jul 30 03:23:05.134470 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:05.134040    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg" podUID="a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" containerName="istio-proxy" containerID="cri-o://feba2c2c6a4375c3d5cda2d2d41e105b9547716e966b1abddfea8afbb6c0d782" gracePeriod=30
Jul 30 03:23:10.386734 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.386705    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:23:10.483032 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.482938    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkkfp\" (UniqueName: \"kubernetes.io/projected/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-kube-api-access-mkkfp\") pod \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") "
Jul 30 03:23:10.483032 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.483014    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istiod-ca-cert\") pod \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") "
Jul 30 03:23:10.483297 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.483039    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-data\") pod \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") "
Jul 30 03:23:10.483297 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.483073    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-envoy\") pod \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") "
Jul 30 03:23:10.483297 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.483111    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-workload-certs\") pod \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") "
Jul 30 03:23:10.483297 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.483154    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-credential-socket\") pod \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") "
Jul 30 03:23:10.483297 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.483206    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-token\") pod \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") "
Jul 30 03:23:10.483297 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.483235    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-podinfo\") pod \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") "
Jul 30 03:23:10.483297 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.483261    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-workload-socket\") pod \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\" (UID: \"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d\") "
Jul 30 03:23:10.483918 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.483491    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" (UID: "a74a1dac-06b5-4ca7-b2a3-a624549d6f1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:23:10.484071 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.483505    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" (UID: "a74a1dac-06b5-4ca7-b2a3-a624549d6f1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:23:10.484131 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.483693    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" (UID: "a74a1dac-06b5-4ca7-b2a3-a624549d6f1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:23:10.484131 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.483613    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-workload-certs\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:23:10.484131 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.484114    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-data" (OuterVolumeSpecName: "istio-data") pod "a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" (UID: "a74a1dac-06b5-4ca7-b2a3-a624549d6f1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:23:10.484333 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.484219    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" (UID: "a74a1dac-06b5-4ca7-b2a3-a624549d6f1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:23:10.485811 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.485782    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-token" (OuterVolumeSpecName: "istio-token") pod "a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" (UID: "a74a1dac-06b5-4ca7-b2a3-a624549d6f1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:23:10.485989 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.485969    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" (UID: "a74a1dac-06b5-4ca7-b2a3-a624549d6f1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:23:10.485989 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.485975    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-kube-api-access-mkkfp" (OuterVolumeSpecName: "kube-api-access-mkkfp") pod "a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" (UID: "a74a1dac-06b5-4ca7-b2a3-a624549d6f1d"). InnerVolumeSpecName "kube-api-access-mkkfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:23:10.486209 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.486192    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" (UID: "a74a1dac-06b5-4ca7-b2a3-a624549d6f1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 30 03:23:10.585345 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.585302    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-envoy\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:23:10.585345 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.585338    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-credential-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:23:10.585345 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.585348    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-token\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:23:10.585345 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.585357    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-podinfo\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:23:10.585635 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.585365    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-workload-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:23:10.585635 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.585374    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkkfp\" (UniqueName: \"kubernetes.io/projected/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-kube-api-access-mkkfp\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:23:10.585635 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.585384    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istiod-ca-cert\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:23:10.585635 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.585392    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d-istio-data\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:23:10.613658 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.613625    2565 generic.go:356] "Generic (PLEG): container finished" podID="a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" containerID="feba2c2c6a4375c3d5cda2d2d41e105b9547716e966b1abddfea8afbb6c0d782" exitCode=0
Jul 30 03:23:10.613819 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.613675    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg" event={"ID":"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d","Type":"ContainerDied","Data":"feba2c2c6a4375c3d5cda2d2d41e105b9547716e966b1abddfea8afbb6c0d782"}
Jul 30 03:23:10.613819 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.613694    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"
Jul 30 03:23:10.613819 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.613745    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg" event={"ID":"a74a1dac-06b5-4ca7-b2a3-a624549d6f1d","Type":"ContainerDied","Data":"b836d5e4d9234c131b9ccd45301276f846bb0b3d04b8b49b988444fefd8f2128"}
Jul 30 03:23:10.613819 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.613767    2565 scope.go:117] "RemoveContainer" containerID="feba2c2c6a4375c3d5cda2d2d41e105b9547716e966b1abddfea8afbb6c0d782"
Jul 30 03:23:10.624106 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.624082    2565 scope.go:117] "RemoveContainer" containerID="1065cde4c485adb6f6a7dfea56cfaab4fb2f5b73ce4b81544d7cee6b6f43f5db"
Jul 30 03:23:10.633490 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.633472    2565 scope.go:117] "RemoveContainer" containerID="feba2c2c6a4375c3d5cda2d2d41e105b9547716e966b1abddfea8afbb6c0d782"
Jul 30 03:23:10.633811 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:23:10.633783    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feba2c2c6a4375c3d5cda2d2d41e105b9547716e966b1abddfea8afbb6c0d782\": container with ID starting with feba2c2c6a4375c3d5cda2d2d41e105b9547716e966b1abddfea8afbb6c0d782 not found: ID does not exist" containerID="feba2c2c6a4375c3d5cda2d2d41e105b9547716e966b1abddfea8afbb6c0d782"
Jul 30 03:23:10.633881 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.633830    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feba2c2c6a4375c3d5cda2d2d41e105b9547716e966b1abddfea8afbb6c0d782"} err="failed to get container status \"feba2c2c6a4375c3d5cda2d2d41e105b9547716e966b1abddfea8afbb6c0d782\": rpc error: code = NotFound desc = could not find container \"feba2c2c6a4375c3d5cda2d2d41e105b9547716e966b1abddfea8afbb6c0d782\": container with ID starting with feba2c2c6a4375c3d5cda2d2d41e105b9547716e966b1abddfea8afbb6c0d782 not found: ID does not exist"
Jul 30 03:23:10.633881 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.633856    2565 scope.go:117] "RemoveContainer" containerID="1065cde4c485adb6f6a7dfea56cfaab4fb2f5b73ce4b81544d7cee6b6f43f5db"
Jul 30 03:23:10.634259 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:23:10.634193    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1065cde4c485adb6f6a7dfea56cfaab4fb2f5b73ce4b81544d7cee6b6f43f5db\": container with ID starting with 1065cde4c485adb6f6a7dfea56cfaab4fb2f5b73ce4b81544d7cee6b6f43f5db not found: ID does not exist" containerID="1065cde4c485adb6f6a7dfea56cfaab4fb2f5b73ce4b81544d7cee6b6f43f5db"
Jul 30 03:23:10.634259 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.634231    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1065cde4c485adb6f6a7dfea56cfaab4fb2f5b73ce4b81544d7cee6b6f43f5db"} err="failed to get container status \"1065cde4c485adb6f6a7dfea56cfaab4fb2f5b73ce4b81544d7cee6b6f43f5db\": rpc error: code = NotFound desc = could not find container \"1065cde4c485adb6f6a7dfea56cfaab4fb2f5b73ce4b81544d7cee6b6f43f5db\": container with ID starting with 1065cde4c485adb6f6a7dfea56cfaab4fb2f5b73ce4b81544d7cee6b6f43f5db not found: ID does not exist"
Jul 30 03:23:10.637283 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.637257    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"]
Jul 30 03:23:10.641697 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.641677    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-64f2f8-openshift-default-5b84b6d4-nv8lg"]
Jul 30 03:23:10.801520 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:10.801476    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" path="/var/lib/kubelet/pods/a74a1dac-06b5-4ca7-b2a3-a624549d6f1d/volumes"
Jul 30 03:23:24.128084 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.128044    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"]
Jul 30 03:23:24.128960 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.128688    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d" podUID="9775762d-34b7-4161-986a-d462a991aff0" containerName="maas-api" containerID="cri-o://e1376c44a8106e4033b107f0606c31cfe598c47afb34a21b72226424e6da8988" gracePeriod=30
Jul 30 03:23:24.408305 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.408281    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"
Jul 30 03:23:24.524463 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.524434    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9775762d-34b7-4161-986a-d462a991aff0-openshift-service-ca\") pod \"9775762d-34b7-4161-986a-d462a991aff0\" (UID: \"9775762d-34b7-4161-986a-d462a991aff0\") "
Jul 30 03:23:24.524662 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.524566    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvlrr\" (UniqueName: \"kubernetes.io/projected/9775762d-34b7-4161-986a-d462a991aff0-kube-api-access-hvlrr\") pod \"9775762d-34b7-4161-986a-d462a991aff0\" (UID: \"9775762d-34b7-4161-986a-d462a991aff0\") "
Jul 30 03:23:24.524662 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.524639    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9775762d-34b7-4161-986a-d462a991aff0-maas-api-tls\") pod \"9775762d-34b7-4161-986a-d462a991aff0\" (UID: \"9775762d-34b7-4161-986a-d462a991aff0\") "
Jul 30 03:23:24.524780 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.524748    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9775762d-34b7-4161-986a-d462a991aff0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9775762d-34b7-4161-986a-d462a991aff0" (UID: "9775762d-34b7-4161-986a-d462a991aff0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:23:24.524968 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.524948    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9775762d-34b7-4161-986a-d462a991aff0-openshift-service-ca\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:23:24.526740 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.526710    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9775762d-34b7-4161-986a-d462a991aff0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9775762d-34b7-4161-986a-d462a991aff0" (UID: "9775762d-34b7-4161-986a-d462a991aff0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:23:24.526847 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.526786    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9775762d-34b7-4161-986a-d462a991aff0-kube-api-access-hvlrr" (OuterVolumeSpecName: "kube-api-access-hvlrr") pod "9775762d-34b7-4161-986a-d462a991aff0" (UID: "9775762d-34b7-4161-986a-d462a991aff0"). InnerVolumeSpecName "kube-api-access-hvlrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:23:24.625739 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.625701    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvlrr\" (UniqueName: \"kubernetes.io/projected/9775762d-34b7-4161-986a-d462a991aff0-kube-api-access-hvlrr\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:23:24.625739 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.625733    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9775762d-34b7-4161-986a-d462a991aff0-maas-api-tls\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:23:24.697528 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.697443    2565 generic.go:356] "Generic (PLEG): container finished" podID="9775762d-34b7-4161-986a-d462a991aff0" containerID="e1376c44a8106e4033b107f0606c31cfe598c47afb34a21b72226424e6da8988" exitCode=0
Jul 30 03:23:24.697528 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.697490    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"
Jul 30 03:23:24.697705 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.697535    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d" event={"ID":"9775762d-34b7-4161-986a-d462a991aff0","Type":"ContainerDied","Data":"e1376c44a8106e4033b107f0606c31cfe598c47afb34a21b72226424e6da8988"}
Jul 30 03:23:24.697705 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.697604    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d" event={"ID":"9775762d-34b7-4161-986a-d462a991aff0","Type":"ContainerDied","Data":"3579cd8c77b1bf5fccb7fd3a442c01c1d338b7e9bd2e1b94e01a676eb07fd716"}
Jul 30 03:23:24.697705 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.697629    2565 scope.go:117] "RemoveContainer" containerID="e1376c44a8106e4033b107f0606c31cfe598c47afb34a21b72226424e6da8988"
Jul 30 03:23:24.711733 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.711708    2565 scope.go:117] "RemoveContainer" containerID="e1376c44a8106e4033b107f0606c31cfe598c47afb34a21b72226424e6da8988"
Jul 30 03:23:24.712066 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:23:24.712048    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1376c44a8106e4033b107f0606c31cfe598c47afb34a21b72226424e6da8988\": container with ID starting with e1376c44a8106e4033b107f0606c31cfe598c47afb34a21b72226424e6da8988 not found: ID does not exist" containerID="e1376c44a8106e4033b107f0606c31cfe598c47afb34a21b72226424e6da8988"
Jul 30 03:23:24.712141 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.712080    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1376c44a8106e4033b107f0606c31cfe598c47afb34a21b72226424e6da8988"} err="failed to get container status \"e1376c44a8106e4033b107f0606c31cfe598c47afb34a21b72226424e6da8988\": rpc error: code = NotFound desc = could not find container \"e1376c44a8106e4033b107f0606c31cfe598c47afb34a21b72226424e6da8988\": container with ID starting with e1376c44a8106e4033b107f0606c31cfe598c47afb34a21b72226424e6da8988 not found: ID does not exist"
Jul 30 03:23:24.720895 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.720841    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"]
Jul 30 03:23:24.724353 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.724332    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cad57-69897d6767-65m7d"]
Jul 30 03:23:24.802821 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:23:24.802789    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9775762d-34b7-4161-986a-d462a991aff0" path="/var/lib/kubelet/pods/9775762d-34b7-4161-986a-d462a991aff0/volumes"
Jul 30 03:24:17.221463 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:17.221423    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"]
Jul 30 03:24:17.221938 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:17.221684    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29" podUID="8d7b8f82-f331-4697-be50-4f724fd68bd0" containerName="istio-proxy" containerID="cri-o://7072c802e1f25b5be38b3ee4fa7a5ded31f8adc8f8052bab80bd8c341b6656e1" gracePeriod=30
Jul 30 03:24:22.481114 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.481084    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:24:22.578985 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.578949    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwx87\" (UniqueName: \"kubernetes.io/projected/8d7b8f82-f331-4697-be50-4f724fd68bd0-kube-api-access-gwx87\") pod \"8d7b8f82-f331-4697-be50-4f724fd68bd0\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") "
Jul 30 03:24:22.579222 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.578999    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-envoy\") pod \"8d7b8f82-f331-4697-be50-4f724fd68bd0\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") "
Jul 30 03:24:22.579222 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.579040    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-token\") pod \"8d7b8f82-f331-4697-be50-4f724fd68bd0\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") "
Jul 30 03:24:22.579222 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.579057    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-podinfo\") pod \"8d7b8f82-f331-4697-be50-4f724fd68bd0\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") "
Jul 30 03:24:22.579222 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.579093    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-workload-certs\") pod \"8d7b8f82-f331-4697-be50-4f724fd68bd0\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") "
Jul 30 03:24:22.579222 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.579117    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-workload-socket\") pod \"8d7b8f82-f331-4697-be50-4f724fd68bd0\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") "
Jul 30 03:24:22.579222 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.579211    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7b8f82-f331-4697-be50-4f724fd68bd0-istiod-ca-cert\") pod \"8d7b8f82-f331-4697-be50-4f724fd68bd0\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") "
Jul 30 03:24:22.579544 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.579242    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-credential-socket\") pod \"8d7b8f82-f331-4697-be50-4f724fd68bd0\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") "
Jul 30 03:24:22.579544 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.579281    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-data\") pod \"8d7b8f82-f331-4697-be50-4f724fd68bd0\" (UID: \"8d7b8f82-f331-4697-be50-4f724fd68bd0\") "
Jul 30 03:24:22.579544 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.579475    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8d7b8f82-f331-4697-be50-4f724fd68bd0" (UID: "8d7b8f82-f331-4697-be50-4f724fd68bd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:24:22.579544 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.579511    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8d7b8f82-f331-4697-be50-4f724fd68bd0" (UID: "8d7b8f82-f331-4697-be50-4f724fd68bd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:24:22.579715 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.579657    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-workload-certs\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:22.579715 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.579677    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-workload-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:22.579715 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.579702    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8d7b8f82-f331-4697-be50-4f724fd68bd0" (UID: "8d7b8f82-f331-4697-be50-4f724fd68bd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:24:22.580009 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.579990    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7b8f82-f331-4697-be50-4f724fd68bd0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8d7b8f82-f331-4697-be50-4f724fd68bd0" (UID: "8d7b8f82-f331-4697-be50-4f724fd68bd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:24:22.580195 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.580147    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-data" (OuterVolumeSpecName: "istio-data") pod "8d7b8f82-f331-4697-be50-4f724fd68bd0" (UID: "8d7b8f82-f331-4697-be50-4f724fd68bd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:24:22.581498 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.581476    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d7b8f82-f331-4697-be50-4f724fd68bd0" (UID: "8d7b8f82-f331-4697-be50-4f724fd68bd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:24:22.581952 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.581933    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8d7b8f82-f331-4697-be50-4f724fd68bd0" (UID: "8d7b8f82-f331-4697-be50-4f724fd68bd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:24:22.582013 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.581957    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7b8f82-f331-4697-be50-4f724fd68bd0-kube-api-access-gwx87" (OuterVolumeSpecName: "kube-api-access-gwx87") pod "8d7b8f82-f331-4697-be50-4f724fd68bd0" (UID: "8d7b8f82-f331-4697-be50-4f724fd68bd0"). InnerVolumeSpecName "kube-api-access-gwx87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:24:22.582013 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.581991    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8d7b8f82-f331-4697-be50-4f724fd68bd0" (UID: "8d7b8f82-f331-4697-be50-4f724fd68bd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 30 03:24:22.680289 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.680255    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-token\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:22.680289 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.680284    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-podinfo\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:22.680289 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.680294    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7b8f82-f331-4697-be50-4f724fd68bd0-istiod-ca-cert\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:22.680535 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.680303    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-credential-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:22.680535 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.680312    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-data\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:22.680535 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.680321    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwx87\" (UniqueName: \"kubernetes.io/projected/8d7b8f82-f331-4697-be50-4f724fd68bd0-kube-api-access-gwx87\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:22.680535 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:22.680330    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7b8f82-f331-4697-be50-4f724fd68bd0-istio-envoy\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:23.049149 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:23.049116    2565 generic.go:356] "Generic (PLEG): container finished" podID="8d7b8f82-f331-4697-be50-4f724fd68bd0" containerID="7072c802e1f25b5be38b3ee4fa7a5ded31f8adc8f8052bab80bd8c341b6656e1" exitCode=0
Jul 30 03:24:23.049335 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:23.049184    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"
Jul 30 03:24:23.049335 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:23.049210    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29" event={"ID":"8d7b8f82-f331-4697-be50-4f724fd68bd0","Type":"ContainerDied","Data":"7072c802e1f25b5be38b3ee4fa7a5ded31f8adc8f8052bab80bd8c341b6656e1"}
Jul 30 03:24:23.049335 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:23.049272    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29" event={"ID":"8d7b8f82-f331-4697-be50-4f724fd68bd0","Type":"ContainerDied","Data":"5600c60e45fbf829c5146ef966382b716518c5ab0306c2a42ae2d51b8da0bc8a"}
Jul 30 03:24:23.049335 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:23.049296    2565 scope.go:117] "RemoveContainer" containerID="7072c802e1f25b5be38b3ee4fa7a5ded31f8adc8f8052bab80bd8c341b6656e1"
Jul 30 03:24:23.058877 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:23.058858    2565 scope.go:117] "RemoveContainer" containerID="18b67542ab75290543d3c786c7f3e8e208feaab026c11d1078d847de16b3327a"
Jul 30 03:24:23.067851 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:23.067830    2565 scope.go:117] "RemoveContainer" containerID="7072c802e1f25b5be38b3ee4fa7a5ded31f8adc8f8052bab80bd8c341b6656e1"
Jul 30 03:24:23.068120 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:24:23.068099    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7072c802e1f25b5be38b3ee4fa7a5ded31f8adc8f8052bab80bd8c341b6656e1\": container with ID starting with 7072c802e1f25b5be38b3ee4fa7a5ded31f8adc8f8052bab80bd8c341b6656e1 not found: ID does not exist" containerID="7072c802e1f25b5be38b3ee4fa7a5ded31f8adc8f8052bab80bd8c341b6656e1"
Jul 30 03:24:23.068224 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:23.068131    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7072c802e1f25b5be38b3ee4fa7a5ded31f8adc8f8052bab80bd8c341b6656e1"} err="failed to get container status \"7072c802e1f25b5be38b3ee4fa7a5ded31f8adc8f8052bab80bd8c341b6656e1\": rpc error: code = NotFound desc = could not find container \"7072c802e1f25b5be38b3ee4fa7a5ded31f8adc8f8052bab80bd8c341b6656e1\": container with ID starting with 7072c802e1f25b5be38b3ee4fa7a5ded31f8adc8f8052bab80bd8c341b6656e1 not found: ID does not exist"
Jul 30 03:24:23.068224 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:23.068148    2565 scope.go:117] "RemoveContainer" containerID="18b67542ab75290543d3c786c7f3e8e208feaab026c11d1078d847de16b3327a"
Jul 30 03:24:23.068402 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:24:23.068385    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b67542ab75290543d3c786c7f3e8e208feaab026c11d1078d847de16b3327a\": container with ID starting with 18b67542ab75290543d3c786c7f3e8e208feaab026c11d1078d847de16b3327a not found: ID does not exist" containerID="18b67542ab75290543d3c786c7f3e8e208feaab026c11d1078d847de16b3327a"
Jul 30 03:24:23.068448 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:23.068407    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b67542ab75290543d3c786c7f3e8e208feaab026c11d1078d847de16b3327a"} err="failed to get container status \"18b67542ab75290543d3c786c7f3e8e208feaab026c11d1078d847de16b3327a\": rpc error: code = NotFound desc = could not find container \"18b67542ab75290543d3c786c7f3e8e208feaab026c11d1078d847de16b3327a\": container with ID starting with 18b67542ab75290543d3c786c7f3e8e208feaab026c11d1078d847de16b3327a not found: ID does not exist"
Jul 30 03:24:23.069128 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:23.069108    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"]
Jul 30 03:24:23.073096 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:23.073066    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2cad57-openshift-default-5bf5cb94db-fsw29"]
Jul 30 03:24:24.800578 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:24.800543    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d7b8f82-f331-4697-be50-4f724fd68bd0" path="/var/lib/kubelet/pods/8d7b8f82-f331-4697-be50-4f724fd68bd0/volumes"
Jul 30 03:24:39.091522 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:39.091483    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"]
Jul 30 03:24:39.092010 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:39.091758    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d" podUID="7c3b9928-470e-40d0-a120-8041725bf0e3" containerName="istio-proxy" containerID="cri-o://9d81bbf20112044fcd51049affaa012282cb7757955a990ddca960ec441baa2f" gracePeriod=30
Jul 30 03:24:44.359556 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.359491    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:24:44.497982 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.497948    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-workload-certs\") pod \"7c3b9928-470e-40d0-a120-8041725bf0e3\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") "
Jul 30 03:24:44.497982 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.497990    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-token\") pod \"7c3b9928-470e-40d0-a120-8041725bf0e3\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") "
Jul 30 03:24:44.498265 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.498020    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-workload-socket\") pod \"7c3b9928-470e-40d0-a120-8041725bf0e3\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") "
Jul 30 03:24:44.498265 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.498060    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-credential-socket\") pod \"7c3b9928-470e-40d0-a120-8041725bf0e3\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") "
Jul 30 03:24:44.498265 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.498078    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7qn4\" (UniqueName: \"kubernetes.io/projected/7c3b9928-470e-40d0-a120-8041725bf0e3-kube-api-access-j7qn4\") pod \"7c3b9928-470e-40d0-a120-8041725bf0e3\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") "
Jul 30 03:24:44.498265 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.498125    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3b9928-470e-40d0-a120-8041725bf0e3-istiod-ca-cert\") pod \"7c3b9928-470e-40d0-a120-8041725bf0e3\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") "
Jul 30 03:24:44.498265 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.498160    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-envoy\") pod \"7c3b9928-470e-40d0-a120-8041725bf0e3\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") "
Jul 30 03:24:44.498265 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.498223    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-data\") pod \"7c3b9928-470e-40d0-a120-8041725bf0e3\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") "
Jul 30 03:24:44.498544 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.498287    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c3b9928-470e-40d0-a120-8041725bf0e3" (UID: "7c3b9928-470e-40d0-a120-8041725bf0e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:24:44.498544 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.498310    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-podinfo\") pod \"7c3b9928-470e-40d0-a120-8041725bf0e3\" (UID: \"7c3b9928-470e-40d0-a120-8041725bf0e3\") "
Jul 30 03:24:44.498544 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.498474    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c3b9928-470e-40d0-a120-8041725bf0e3" (UID: "7c3b9928-470e-40d0-a120-8041725bf0e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:24:44.498544 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.498492    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c3b9928-470e-40d0-a120-8041725bf0e3" (UID: "7c3b9928-470e-40d0-a120-8041725bf0e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:24:44.498742 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.498579    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3b9928-470e-40d0-a120-8041725bf0e3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c3b9928-470e-40d0-a120-8041725bf0e3" (UID: "7c3b9928-470e-40d0-a120-8041725bf0e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:24:44.498742 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.498701    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-workload-certs\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:44.498742 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.498721    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-workload-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:44.498742 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.498735    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-credential-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:44.498936 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.498748    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3b9928-470e-40d0-a120-8041725bf0e3-istiod-ca-cert\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:44.499515 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.499489    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c3b9928-470e-40d0-a120-8041725bf0e3" (UID: "7c3b9928-470e-40d0-a120-8041725bf0e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:24:44.500640 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.500608    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c3b9928-470e-40d0-a120-8041725bf0e3" (UID: "7c3b9928-470e-40d0-a120-8041725bf0e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:24:44.500763 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.500706    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3b9928-470e-40d0-a120-8041725bf0e3-kube-api-access-j7qn4" (OuterVolumeSpecName: "kube-api-access-j7qn4") pod "7c3b9928-470e-40d0-a120-8041725bf0e3" (UID: "7c3b9928-470e-40d0-a120-8041725bf0e3"). InnerVolumeSpecName "kube-api-access-j7qn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:24:44.501009 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.500989    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c3b9928-470e-40d0-a120-8041725bf0e3" (UID: "7c3b9928-470e-40d0-a120-8041725bf0e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 30 03:24:44.501086 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.500999    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c3b9928-470e-40d0-a120-8041725bf0e3" (UID: "7c3b9928-470e-40d0-a120-8041725bf0e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:24:44.599633 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.599599    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7qn4\" (UniqueName: \"kubernetes.io/projected/7c3b9928-470e-40d0-a120-8041725bf0e3-kube-api-access-j7qn4\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:44.599633 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.599628    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-envoy\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:44.599633 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.599637    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-data\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:44.599633 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.599645    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-podinfo\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:44.599902 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:44.599653    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3b9928-470e-40d0-a120-8041725bf0e3-istio-token\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:45.192799 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:45.192702    2565 generic.go:356] "Generic (PLEG): container finished" podID="7c3b9928-470e-40d0-a120-8041725bf0e3" containerID="9d81bbf20112044fcd51049affaa012282cb7757955a990ddca960ec441baa2f" exitCode=0
Jul 30 03:24:45.192799 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:45.192756    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"
Jul 30 03:24:45.192799 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:45.192784    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d" event={"ID":"7c3b9928-470e-40d0-a120-8041725bf0e3","Type":"ContainerDied","Data":"9d81bbf20112044fcd51049affaa012282cb7757955a990ddca960ec441baa2f"}
Jul 30 03:24:45.193069 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:45.192838    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d" event={"ID":"7c3b9928-470e-40d0-a120-8041725bf0e3","Type":"ContainerDied","Data":"c6e63bf780690b31c5e469337ee711f8b02b1467b70399f9db4d57d60677e0c6"}
Jul 30 03:24:45.193069 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:45.192854    2565 scope.go:117] "RemoveContainer" containerID="9d81bbf20112044fcd51049affaa012282cb7757955a990ddca960ec441baa2f"
Jul 30 03:24:45.203254 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:45.203237    2565 scope.go:117] "RemoveContainer" containerID="3b5d1e6142b7c5ebbe94383cfa317d3a14f1b4d6f0e5b6276f567856f71a8355"
Jul 30 03:24:45.218249 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:45.213187    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"]
Jul 30 03:24:45.220397 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:45.220293    2565 scope.go:117] "RemoveContainer" containerID="9d81bbf20112044fcd51049affaa012282cb7757955a990ddca960ec441baa2f"
Jul 30 03:24:45.220696 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:24:45.220663    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d81bbf20112044fcd51049affaa012282cb7757955a990ddca960ec441baa2f\": container with ID starting with 9d81bbf20112044fcd51049affaa012282cb7757955a990ddca960ec441baa2f not found: ID does not exist" containerID="9d81bbf20112044fcd51049affaa012282cb7757955a990ddca960ec441baa2f"
Jul 30 03:24:45.220812 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:45.220704    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d81bbf20112044fcd51049affaa012282cb7757955a990ddca960ec441baa2f"} err="failed to get container status \"9d81bbf20112044fcd51049affaa012282cb7757955a990ddca960ec441baa2f\": rpc error: code = NotFound desc = could not find container \"9d81bbf20112044fcd51049affaa012282cb7757955a990ddca960ec441baa2f\": container with ID starting with 9d81bbf20112044fcd51049affaa012282cb7757955a990ddca960ec441baa2f not found: ID does not exist"
Jul 30 03:24:45.220812 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:45.220723    2565 scope.go:117] "RemoveContainer" containerID="3b5d1e6142b7c5ebbe94383cfa317d3a14f1b4d6f0e5b6276f567856f71a8355"
Jul 30 03:24:45.221056 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:24:45.221038    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5d1e6142b7c5ebbe94383cfa317d3a14f1b4d6f0e5b6276f567856f71a8355\": container with ID starting with 3b5d1e6142b7c5ebbe94383cfa317d3a14f1b4d6f0e5b6276f567856f71a8355 not found: ID does not exist" containerID="3b5d1e6142b7c5ebbe94383cfa317d3a14f1b4d6f0e5b6276f567856f71a8355"
Jul 30 03:24:45.221119 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:45.221063    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5d1e6142b7c5ebbe94383cfa317d3a14f1b4d6f0e5b6276f567856f71a8355"} err="failed to get container status \"3b5d1e6142b7c5ebbe94383cfa317d3a14f1b4d6f0e5b6276f567856f71a8355\": rpc error: code = NotFound desc = could not find container \"3b5d1e6142b7c5ebbe94383cfa317d3a14f1b4d6f0e5b6276f567856f71a8355\": container with ID starting with 3b5d1e6142b7c5ebbe94383cfa317d3a14f1b4d6f0e5b6276f567856f71a8355 not found: ID does not exist"
Jul 30 03:24:45.223465 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:45.223442    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f58ab6-openshift-default-775bd4c97b-pmw9d"]
Jul 30 03:24:46.801273 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:46.801236    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c3b9928-470e-40d0-a120-8041725bf0e3" path="/var/lib/kubelet/pods/7c3b9928-470e-40d0-a120-8041725bf0e3/volumes"
Jul 30 03:24:50.022013 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:50.021977    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"]
Jul 30 03:24:50.022559 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:50.022253    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms" podUID="d4fb5b4d-5973-4860-bbf3-854e36472540" containerName="maas-api" containerID="cri-o://3d3296000e1c82907aa856cdbfa804519f3021c6aea16c2cf6ff67824b559a66" gracePeriod=30
Jul 30 03:24:50.225117 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:50.225086    2565 generic.go:356] "Generic (PLEG): container finished" podID="d4fb5b4d-5973-4860-bbf3-854e36472540" containerID="3d3296000e1c82907aa856cdbfa804519f3021c6aea16c2cf6ff67824b559a66" exitCode=0
Jul 30 03:24:50.225301 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:50.225193    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms" event={"ID":"d4fb5b4d-5973-4860-bbf3-854e36472540","Type":"ContainerDied","Data":"3d3296000e1c82907aa856cdbfa804519f3021c6aea16c2cf6ff67824b559a66"}
Jul 30 03:24:50.287419 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:50.287341    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"
Jul 30 03:24:50.453452 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:50.453409    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fb5b4d-5973-4860-bbf3-854e36472540-openshift-service-ca\") pod \"d4fb5b4d-5973-4860-bbf3-854e36472540\" (UID: \"d4fb5b4d-5973-4860-bbf3-854e36472540\") "
Jul 30 03:24:50.453655 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:50.453462    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpj8q\" (UniqueName: \"kubernetes.io/projected/d4fb5b4d-5973-4860-bbf3-854e36472540-kube-api-access-vpj8q\") pod \"d4fb5b4d-5973-4860-bbf3-854e36472540\" (UID: \"d4fb5b4d-5973-4860-bbf3-854e36472540\") "
Jul 30 03:24:50.453655 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:50.453509    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fb5b4d-5973-4860-bbf3-854e36472540-maas-api-tls\") pod \"d4fb5b4d-5973-4860-bbf3-854e36472540\" (UID: \"d4fb5b4d-5973-4860-bbf3-854e36472540\") "
Jul 30 03:24:50.453804 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:50.453777    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fb5b4d-5973-4860-bbf3-854e36472540-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d4fb5b4d-5973-4860-bbf3-854e36472540" (UID: "d4fb5b4d-5973-4860-bbf3-854e36472540"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:24:50.455731 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:50.455700    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4fb5b4d-5973-4860-bbf3-854e36472540-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d4fb5b4d-5973-4860-bbf3-854e36472540" (UID: "d4fb5b4d-5973-4860-bbf3-854e36472540"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:24:50.455868 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:50.455845    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fb5b4d-5973-4860-bbf3-854e36472540-kube-api-access-vpj8q" (OuterVolumeSpecName: "kube-api-access-vpj8q") pod "d4fb5b4d-5973-4860-bbf3-854e36472540" (UID: "d4fb5b4d-5973-4860-bbf3-854e36472540"). InnerVolumeSpecName "kube-api-access-vpj8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:24:50.554824 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:50.554792    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fb5b4d-5973-4860-bbf3-854e36472540-openshift-service-ca\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:50.554824 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:50.554821    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpj8q\" (UniqueName: \"kubernetes.io/projected/d4fb5b4d-5973-4860-bbf3-854e36472540-kube-api-access-vpj8q\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:50.555023 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:50.554831    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4fb5b4d-5973-4860-bbf3-854e36472540-maas-api-tls\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:24:51.239476 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:51.239386    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms" event={"ID":"d4fb5b4d-5973-4860-bbf3-854e36472540","Type":"ContainerDied","Data":"009fa4d05e4627b761842c3f04c7eb71cf6811319d1e2cc6db08e42c3fdbd146"}
Jul 30 03:24:51.239476 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:51.239439    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"
Jul 30 03:24:51.239476 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:51.239474    2565 scope.go:117] "RemoveContainer" containerID="3d3296000e1c82907aa856cdbfa804519f3021c6aea16c2cf6ff67824b559a66"
Jul 30 03:24:51.257529 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:51.257497    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"]
Jul 30 03:24:51.261458 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:51.261429    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-da5459-7ddbc8657c-469ms"]
Jul 30 03:24:52.801347 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:24:52.801308    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4fb5b4d-5973-4860-bbf3-854e36472540" path="/var/lib/kubelet/pods/d4fb5b4d-5973-4860-bbf3-854e36472540/volumes"
Jul 30 03:25:01.013744 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:01.013709    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"]
Jul 30 03:25:01.014330 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:01.013947    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p" podUID="8ba45532-1c80-4328-b6d2-c73ff607f539" containerName="istio-proxy" containerID="cri-o://e308bca0c77308ff0b0ebbc0b92d78113e69ec80364f3afb8db4f587a49b6ca0" gracePeriod=30
Jul 30 03:25:03.566430 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:03.566380    2565 patch_prober.go:28] interesting pod/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 30 03:25:03.566817 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:03.566465    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p" podUID="8ba45532-1c80-4328-b6d2-c73ff607f539" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 03:25:06.280887 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.280821    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:25:06.308124 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.308093    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7vql\" (UniqueName: \"kubernetes.io/projected/8ba45532-1c80-4328-b6d2-c73ff607f539-kube-api-access-c7vql\") pod \"8ba45532-1c80-4328-b6d2-c73ff607f539\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") "
Jul 30 03:25:06.308335 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.308150    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-data\") pod \"8ba45532-1c80-4328-b6d2-c73ff607f539\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") "
Jul 30 03:25:06.308335 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.308205    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba45532-1c80-4328-b6d2-c73ff607f539-istiod-ca-cert\") pod \"8ba45532-1c80-4328-b6d2-c73ff607f539\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") "
Jul 30 03:25:06.308335 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.308253    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-podinfo\") pod \"8ba45532-1c80-4328-b6d2-c73ff607f539\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") "
Jul 30 03:25:06.308335 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.308308    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-workload-socket\") pod \"8ba45532-1c80-4328-b6d2-c73ff607f539\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") "
Jul 30 03:25:06.308582 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.308365    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-workload-certs\") pod \"8ba45532-1c80-4328-b6d2-c73ff607f539\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") "
Jul 30 03:25:06.308582 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.308403    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-envoy\") pod \"8ba45532-1c80-4328-b6d2-c73ff607f539\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") "
Jul 30 03:25:06.308582 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.308443    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-token\") pod \"8ba45532-1c80-4328-b6d2-c73ff607f539\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") "
Jul 30 03:25:06.308582 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.308473    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-credential-socket\") pod \"8ba45532-1c80-4328-b6d2-c73ff607f539\" (UID: \"8ba45532-1c80-4328-b6d2-c73ff607f539\") "
Jul 30 03:25:06.308804 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.308658    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba45532-1c80-4328-b6d2-c73ff607f539-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8ba45532-1c80-4328-b6d2-c73ff607f539" (UID: "8ba45532-1c80-4328-b6d2-c73ff607f539"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:25:06.308864 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.308778    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8ba45532-1c80-4328-b6d2-c73ff607f539" (UID: "8ba45532-1c80-4328-b6d2-c73ff607f539"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:25:06.309311 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.308973    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8ba45532-1c80-4328-b6d2-c73ff607f539" (UID: "8ba45532-1c80-4328-b6d2-c73ff607f539"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:25:06.309311 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.309038    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-data" (OuterVolumeSpecName: "istio-data") pod "8ba45532-1c80-4328-b6d2-c73ff607f539" (UID: "8ba45532-1c80-4328-b6d2-c73ff607f539"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:25:06.309311 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.309152    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8ba45532-1c80-4328-b6d2-c73ff607f539" (UID: "8ba45532-1c80-4328-b6d2-c73ff607f539"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:25:06.311076 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.311052    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba45532-1c80-4328-b6d2-c73ff607f539-kube-api-access-c7vql" (OuterVolumeSpecName: "kube-api-access-c7vql") pod "8ba45532-1c80-4328-b6d2-c73ff607f539" (UID: "8ba45532-1c80-4328-b6d2-c73ff607f539"). InnerVolumeSpecName "kube-api-access-c7vql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:25:06.311192 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.311140    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-token" (OuterVolumeSpecName: "istio-token") pod "8ba45532-1c80-4328-b6d2-c73ff607f539" (UID: "8ba45532-1c80-4328-b6d2-c73ff607f539"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:25:06.311250 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.311198    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8ba45532-1c80-4328-b6d2-c73ff607f539" (UID: "8ba45532-1c80-4328-b6d2-c73ff607f539"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:25:06.311453 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.311430    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8ba45532-1c80-4328-b6d2-c73ff607f539" (UID: "8ba45532-1c80-4328-b6d2-c73ff607f539"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 30 03:25:06.336576 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.336539    2565 generic.go:356] "Generic (PLEG): container finished" podID="8ba45532-1c80-4328-b6d2-c73ff607f539" containerID="e308bca0c77308ff0b0ebbc0b92d78113e69ec80364f3afb8db4f587a49b6ca0" exitCode=0
Jul 30 03:25:06.336714 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.336588    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"
Jul 30 03:25:06.336714 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.336611    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p" event={"ID":"8ba45532-1c80-4328-b6d2-c73ff607f539","Type":"ContainerDied","Data":"e308bca0c77308ff0b0ebbc0b92d78113e69ec80364f3afb8db4f587a49b6ca0"}
Jul 30 03:25:06.336714 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.336708    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p" event={"ID":"8ba45532-1c80-4328-b6d2-c73ff607f539","Type":"ContainerDied","Data":"207ab3d76bf467f064d1f10e4ebb8775535d0662374d3afc5e0f2e5e9531a27d"}
Jul 30 03:25:06.336818 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.336738    2565 scope.go:117] "RemoveContainer" containerID="e308bca0c77308ff0b0ebbc0b92d78113e69ec80364f3afb8db4f587a49b6ca0"
Jul 30 03:25:06.348139 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.348117    2565 scope.go:117] "RemoveContainer" containerID="cb46a97547962b06b315845a17164e37c1a11319b71f6e39c820d66313bf79f5"
Jul 30 03:25:06.361967 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.361937    2565 scope.go:117] "RemoveContainer" containerID="e308bca0c77308ff0b0ebbc0b92d78113e69ec80364f3afb8db4f587a49b6ca0"
Jul 30 03:25:06.362689 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.362645    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"]
Jul 30 03:25:06.362925 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:25:06.362895    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e308bca0c77308ff0b0ebbc0b92d78113e69ec80364f3afb8db4f587a49b6ca0\": container with ID starting with e308bca0c77308ff0b0ebbc0b92d78113e69ec80364f3afb8db4f587a49b6ca0 not found: ID does not exist" containerID="e308bca0c77308ff0b0ebbc0b92d78113e69ec80364f3afb8db4f587a49b6ca0"
Jul 30 03:25:06.363059 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.362935    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e308bca0c77308ff0b0ebbc0b92d78113e69ec80364f3afb8db4f587a49b6ca0"} err="failed to get container status \"e308bca0c77308ff0b0ebbc0b92d78113e69ec80364f3afb8db4f587a49b6ca0\": rpc error: code = NotFound desc = could not find container \"e308bca0c77308ff0b0ebbc0b92d78113e69ec80364f3afb8db4f587a49b6ca0\": container with ID starting with e308bca0c77308ff0b0ebbc0b92d78113e69ec80364f3afb8db4f587a49b6ca0 not found: ID does not exist"
Jul 30 03:25:06.363059 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.362960    2565 scope.go:117] "RemoveContainer" containerID="cb46a97547962b06b315845a17164e37c1a11319b71f6e39c820d66313bf79f5"
Jul 30 03:25:06.363288 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:25:06.363266    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb46a97547962b06b315845a17164e37c1a11319b71f6e39c820d66313bf79f5\": container with ID starting with cb46a97547962b06b315845a17164e37c1a11319b71f6e39c820d66313bf79f5 not found: ID does not exist" containerID="cb46a97547962b06b315845a17164e37c1a11319b71f6e39c820d66313bf79f5"
Jul 30 03:25:06.363354 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.363296    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb46a97547962b06b315845a17164e37c1a11319b71f6e39c820d66313bf79f5"} err="failed to get container status \"cb46a97547962b06b315845a17164e37c1a11319b71f6e39c820d66313bf79f5\": rpc error: code = NotFound desc = could not find container \"cb46a97547962b06b315845a17164e37c1a11319b71f6e39c820d66313bf79f5\": container with ID starting with cb46a97547962b06b315845a17164e37c1a11319b71f6e39c820d66313bf79f5 not found: ID does not exist"
Jul 30 03:25:06.366664 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.366642    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-da5459-openshift-default-564b7c557b-ccr8p"]
Jul 30 03:25:06.370476 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:25:06.370450    2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba45532_1c80_4328_b6d2_c73ff607f539.slice\": RecentStats: unable to find data in memory cache]"
Jul 30 03:25:06.410084 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.410042    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-podinfo\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:06.410084 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.410080    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-workload-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:06.410084 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.410089    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-workload-certs\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:06.410084 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.410100    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-envoy\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:06.410376 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.410109    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-token\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:06.410376 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.410116    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-credential-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:06.410376 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.410125    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7vql\" (UniqueName: \"kubernetes.io/projected/8ba45532-1c80-4328-b6d2-c73ff607f539-kube-api-access-c7vql\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:06.410376 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.410134    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba45532-1c80-4328-b6d2-c73ff607f539-istio-data\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:06.410376 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.410142    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba45532-1c80-4328-b6d2-c73ff607f539-istiod-ca-cert\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:06.800425 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:06.800387    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ba45532-1c80-4328-b6d2-c73ff607f539" path="/var/lib/kubelet/pods/8ba45532-1c80-4328-b6d2-c73ff607f539/volumes"
Jul 30 03:25:11.767622 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:11.767584    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"]
Jul 30 03:25:11.769064 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:11.769024    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh" podUID="990d6ed8-feb1-455b-ba3a-a5243bbd63fc" containerName="maas-api" containerID="cri-o://0e7e445c62680b0d5158a87a2947e6a831cd02d53c3b4133262d63feb7fe6f4c" gracePeriod=30
Jul 30 03:25:11.794980 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:11.794954    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f2a805-29756355-tg6qg"]
Jul 30 03:25:11.797235 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:11.797211    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f2a805-29756355-tg6qg"]
Jul 30 03:25:12.040945 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.040912    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"
Jul 30 03:25:12.164008 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.163970    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-openshift-service-ca\") pod \"990d6ed8-feb1-455b-ba3a-a5243bbd63fc\" (UID: \"990d6ed8-feb1-455b-ba3a-a5243bbd63fc\") "
Jul 30 03:25:12.164157 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.164019    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-maas-api-tls\") pod \"990d6ed8-feb1-455b-ba3a-a5243bbd63fc\" (UID: \"990d6ed8-feb1-455b-ba3a-a5243bbd63fc\") "
Jul 30 03:25:12.164157 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.164057    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xm6r\" (UniqueName: \"kubernetes.io/projected/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-kube-api-access-7xm6r\") pod \"990d6ed8-feb1-455b-ba3a-a5243bbd63fc\" (UID: \"990d6ed8-feb1-455b-ba3a-a5243bbd63fc\") "
Jul 30 03:25:12.164413 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.164387    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "990d6ed8-feb1-455b-ba3a-a5243bbd63fc" (UID: "990d6ed8-feb1-455b-ba3a-a5243bbd63fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:25:12.166406 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.166375    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "990d6ed8-feb1-455b-ba3a-a5243bbd63fc" (UID: "990d6ed8-feb1-455b-ba3a-a5243bbd63fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:25:12.166530 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.166454    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-kube-api-access-7xm6r" (OuterVolumeSpecName: "kube-api-access-7xm6r") pod "990d6ed8-feb1-455b-ba3a-a5243bbd63fc" (UID: "990d6ed8-feb1-455b-ba3a-a5243bbd63fc"). InnerVolumeSpecName "kube-api-access-7xm6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:25:12.265377 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.265331    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-openshift-service-ca\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:12.265377 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.265367    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-maas-api-tls\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:12.265377 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.265378    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xm6r\" (UniqueName: \"kubernetes.io/projected/990d6ed8-feb1-455b-ba3a-a5243bbd63fc-kube-api-access-7xm6r\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:12.377023 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.376986    2565 generic.go:356] "Generic (PLEG): container finished" podID="990d6ed8-feb1-455b-ba3a-a5243bbd63fc" containerID="0e7e445c62680b0d5158a87a2947e6a831cd02d53c3b4133262d63feb7fe6f4c" exitCode=0
Jul 30 03:25:12.377254 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.377051    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"
Jul 30 03:25:12.377254 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.377077    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh" event={"ID":"990d6ed8-feb1-455b-ba3a-a5243bbd63fc","Type":"ContainerDied","Data":"0e7e445c62680b0d5158a87a2947e6a831cd02d53c3b4133262d63feb7fe6f4c"}
Jul 30 03:25:12.377254 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.377154    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh" event={"ID":"990d6ed8-feb1-455b-ba3a-a5243bbd63fc","Type":"ContainerDied","Data":"7c2443ef636550934af4cf65d8f9bff75363f1ba2f9a57f0dacbe4d7b39cd6cd"}
Jul 30 03:25:12.377254 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.377206    2565 scope.go:117] "RemoveContainer" containerID="0e7e445c62680b0d5158a87a2947e6a831cd02d53c3b4133262d63feb7fe6f4c"
Jul 30 03:25:12.387625 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.387604    2565 scope.go:117] "RemoveContainer" containerID="0e7e445c62680b0d5158a87a2947e6a831cd02d53c3b4133262d63feb7fe6f4c"
Jul 30 03:25:12.387916 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:25:12.387894    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e7e445c62680b0d5158a87a2947e6a831cd02d53c3b4133262d63feb7fe6f4c\": container with ID starting with 0e7e445c62680b0d5158a87a2947e6a831cd02d53c3b4133262d63feb7fe6f4c not found: ID does not exist" containerID="0e7e445c62680b0d5158a87a2947e6a831cd02d53c3b4133262d63feb7fe6f4c"
Jul 30 03:25:12.387974 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.387942    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e7e445c62680b0d5158a87a2947e6a831cd02d53c3b4133262d63feb7fe6f4c"} err="failed to get container status \"0e7e445c62680b0d5158a87a2947e6a831cd02d53c3b4133262d63feb7fe6f4c\": rpc error: code = NotFound desc = could not find container \"0e7e445c62680b0d5158a87a2947e6a831cd02d53c3b4133262d63feb7fe6f4c\": container with ID starting with 0e7e445c62680b0d5158a87a2947e6a831cd02d53c3b4133262d63feb7fe6f4c not found: ID does not exist"
Jul 30 03:25:12.400507 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.400475    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"]
Jul 30 03:25:12.405826 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.405802    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f2a805-746b797b68-2s8dh"]
Jul 30 03:25:12.800191 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.800133    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9639c0c2-fd3a-4309-9ee1-996060a84ead" path="/var/lib/kubelet/pods/9639c0c2-fd3a-4309-9ee1-996060a84ead/volumes"
Jul 30 03:25:12.800570 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:12.800517    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="990d6ed8-feb1-455b-ba3a-a5243bbd63fc" path="/var/lib/kubelet/pods/990d6ed8-feb1-455b-ba3a-a5243bbd63fc/volumes"
Jul 30 03:25:22.916391 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:22.916356    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"]
Jul 30 03:25:22.916854 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:22.916570    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5" podUID="02bb64dc-de3e-4e34-b139-d6510f2eb893" containerName="istio-proxy" containerID="cri-o://30a3a891126007226b019938fd6f6bb1ddb5531bc2a351bc67ff4a850f154072" gracePeriod=30
Jul 30 03:25:26.654767 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:26.654727    2565 patch_prober.go:28] interesting pod/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 30 03:25:26.655141 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:26.654790    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5" podUID="02bb64dc-de3e-4e34-b139-d6510f2eb893" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 03:25:28.174979 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.174953    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:25:28.314244 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.314215    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-token\") pod \"02bb64dc-de3e-4e34-b139-d6510f2eb893\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") "
Jul 30 03:25:28.314429 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.314277    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-credential-socket\") pod \"02bb64dc-de3e-4e34-b139-d6510f2eb893\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") "
Jul 30 03:25:28.314429 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.314309    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-podinfo\") pod \"02bb64dc-de3e-4e34-b139-d6510f2eb893\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") "
Jul 30 03:25:28.314429 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.314342    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-data\") pod \"02bb64dc-de3e-4e34-b139-d6510f2eb893\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") "
Jul 30 03:25:28.314429 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.314380    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpgpc\" (UniqueName: \"kubernetes.io/projected/02bb64dc-de3e-4e34-b139-d6510f2eb893-kube-api-access-cpgpc\") pod \"02bb64dc-de3e-4e34-b139-d6510f2eb893\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") "
Jul 30 03:25:28.314649 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.314446    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-workload-certs\") pod \"02bb64dc-de3e-4e34-b139-d6510f2eb893\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") "
Jul 30 03:25:28.314649 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.314603    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02bb64dc-de3e-4e34-b139-d6510f2eb893-istiod-ca-cert\") pod \"02bb64dc-de3e-4e34-b139-d6510f2eb893\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") "
Jul 30 03:25:28.314758 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.314662    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-envoy\") pod \"02bb64dc-de3e-4e34-b139-d6510f2eb893\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") "
Jul 30 03:25:28.314758 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.314645    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "02bb64dc-de3e-4e34-b139-d6510f2eb893" (UID: "02bb64dc-de3e-4e34-b139-d6510f2eb893"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:25:28.314758 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.314686    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-workload-socket\") pod \"02bb64dc-de3e-4e34-b139-d6510f2eb893\" (UID: \"02bb64dc-de3e-4e34-b139-d6510f2eb893\") "
Jul 30 03:25:28.314758 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.314735    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "02bb64dc-de3e-4e34-b139-d6510f2eb893" (UID: "02bb64dc-de3e-4e34-b139-d6510f2eb893"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:25:28.314971 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.314904    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02bb64dc-de3e-4e34-b139-d6510f2eb893-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "02bb64dc-de3e-4e34-b139-d6510f2eb893" (UID: "02bb64dc-de3e-4e34-b139-d6510f2eb893"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:25:28.315191 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.315110    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-workload-certs\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:28.315191 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.315135    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02bb64dc-de3e-4e34-b139-d6510f2eb893-istiod-ca-cert\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:28.315191 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.315148    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-credential-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:28.315403 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.315196    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "02bb64dc-de3e-4e34-b139-d6510f2eb893" (UID: "02bb64dc-de3e-4e34-b139-d6510f2eb893"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:25:28.315496 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.315470    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-data" (OuterVolumeSpecName: "istio-data") pod "02bb64dc-de3e-4e34-b139-d6510f2eb893" (UID: "02bb64dc-de3e-4e34-b139-d6510f2eb893"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:25:28.316710 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.316688    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "02bb64dc-de3e-4e34-b139-d6510f2eb893" (UID: "02bb64dc-de3e-4e34-b139-d6510f2eb893"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:25:28.316845 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.316801    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "02bb64dc-de3e-4e34-b139-d6510f2eb893" (UID: "02bb64dc-de3e-4e34-b139-d6510f2eb893"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 30 03:25:28.317118 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.317103    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-token" (OuterVolumeSpecName: "istio-token") pod "02bb64dc-de3e-4e34-b139-d6510f2eb893" (UID: "02bb64dc-de3e-4e34-b139-d6510f2eb893"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:25:28.317213 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.317192    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bb64dc-de3e-4e34-b139-d6510f2eb893-kube-api-access-cpgpc" (OuterVolumeSpecName: "kube-api-access-cpgpc") pod "02bb64dc-de3e-4e34-b139-d6510f2eb893" (UID: "02bb64dc-de3e-4e34-b139-d6510f2eb893"). InnerVolumeSpecName "kube-api-access-cpgpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:25:28.416226 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.416190    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-envoy\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:28.416226 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.416219    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-workload-socket\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:28.416226 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.416229    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-token\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:28.416226 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.416237    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-podinfo\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:28.416495 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.416245    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02bb64dc-de3e-4e34-b139-d6510f2eb893-istio-data\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:28.416495 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.416253    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpgpc\" (UniqueName: \"kubernetes.io/projected/02bb64dc-de3e-4e34-b139-d6510f2eb893-kube-api-access-cpgpc\") on node \"ip-10-0-132-89.ec2.internal\" DevicePath \"\""
Jul 30 03:25:28.470963 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.470927    2565 generic.go:356] "Generic (PLEG): container finished" podID="02bb64dc-de3e-4e34-b139-d6510f2eb893" containerID="30a3a891126007226b019938fd6f6bb1ddb5531bc2a351bc67ff4a850f154072" exitCode=0
Jul 30 03:25:28.471119 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.470977    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"
Jul 30 03:25:28.471119 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.471010    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5" event={"ID":"02bb64dc-de3e-4e34-b139-d6510f2eb893","Type":"ContainerDied","Data":"30a3a891126007226b019938fd6f6bb1ddb5531bc2a351bc67ff4a850f154072"}
Jul 30 03:25:28.471119 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.471064    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5" event={"ID":"02bb64dc-de3e-4e34-b139-d6510f2eb893","Type":"ContainerDied","Data":"b0b25d1ab0a74263acbfcab9a7a9329f2ad9dd74c7141b27d728b66fc9acdbcf"}
Jul 30 03:25:28.471119 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.471088    2565 scope.go:117] "RemoveContainer" containerID="30a3a891126007226b019938fd6f6bb1ddb5531bc2a351bc67ff4a850f154072"
Jul 30 03:25:28.481058 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.481038    2565 scope.go:117] "RemoveContainer" containerID="1866844b5c2fd1b790ec9cb765f9666484a08bfcc6345763495085fcab2bb331"
Jul 30 03:25:28.490042 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.490024    2565 scope.go:117] "RemoveContainer" containerID="30a3a891126007226b019938fd6f6bb1ddb5531bc2a351bc67ff4a850f154072"
Jul 30 03:25:28.490345 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:25:28.490323    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a3a891126007226b019938fd6f6bb1ddb5531bc2a351bc67ff4a850f154072\": container with ID starting with 30a3a891126007226b019938fd6f6bb1ddb5531bc2a351bc67ff4a850f154072 not found: ID does not exist" containerID="30a3a891126007226b019938fd6f6bb1ddb5531bc2a351bc67ff4a850f154072"
Jul 30 03:25:28.490398 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.490354    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a3a891126007226b019938fd6f6bb1ddb5531bc2a351bc67ff4a850f154072"} err="failed to get container status \"30a3a891126007226b019938fd6f6bb1ddb5531bc2a351bc67ff4a850f154072\": rpc error: code = NotFound desc = could not find container \"30a3a891126007226b019938fd6f6bb1ddb5531bc2a351bc67ff4a850f154072\": container with ID starting with 30a3a891126007226b019938fd6f6bb1ddb5531bc2a351bc67ff4a850f154072 not found: ID does not exist"
Jul 30 03:25:28.490398 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.490370    2565 scope.go:117] "RemoveContainer" containerID="1866844b5c2fd1b790ec9cb765f9666484a08bfcc6345763495085fcab2bb331"
Jul 30 03:25:28.490674 ip-10-0-132-89 kubenswrapper[2565]: E0730 03:25:28.490649    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1866844b5c2fd1b790ec9cb765f9666484a08bfcc6345763495085fcab2bb331\": container with ID starting with 1866844b5c2fd1b790ec9cb765f9666484a08bfcc6345763495085fcab2bb331 not found: ID does not exist" containerID="1866844b5c2fd1b790ec9cb765f9666484a08bfcc6345763495085fcab2bb331"
Jul 30 03:25:28.490729 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.490685    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1866844b5c2fd1b790ec9cb765f9666484a08bfcc6345763495085fcab2bb331"} err="failed to get container status \"1866844b5c2fd1b790ec9cb765f9666484a08bfcc6345763495085fcab2bb331\": rpc error: code = NotFound desc = could not find container \"1866844b5c2fd1b790ec9cb765f9666484a08bfcc6345763495085fcab2bb331\": container with ID starting with 1866844b5c2fd1b790ec9cb765f9666484a08bfcc6345763495085fcab2bb331 not found: ID does not exist"
Jul 30 03:25:28.494370 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.494338    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"]
Jul 30 03:25:28.502194 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.500338    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-f2a805-openshift-default-587fb544dd-6ftk5"]
Jul 30 03:25:28.787048 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.786972    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5kntq_cf4f1044-78f4-4086-aac0-406665379140/manager/0.log"
Jul 30 03:25:28.801261 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.801224    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02bb64dc-de3e-4e34-b139-d6510f2eb893" path="/var/lib/kubelet/pods/02bb64dc-de3e-4e34-b139-d6510f2eb893/volumes"
Jul 30 03:25:28.901476 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:28.901435    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-x28m9_6e9e9962-00bd-4256-a254-276c940b6d80/manager/0.log"
Jul 30 03:25:29.012938 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:29.012906    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6899b54cb8-4k7tc_4b1585e2-ab06-4fc4-ad31-c3cf9912ea2b/manager/0.log"
Jul 30 03:25:29.587430 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:29.587398    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-684d7c5d99-sc8pn_93cde41a-963f-4f29-814d-4c909d54efa3/copy-manifests/0.log"
Jul 30 03:25:29.593909 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:29.593880    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-684d7c5d99-sc8pn_93cde41a-963f-4f29-814d-4c909d54efa3/manager/0.log"
Jul 30 03:25:29.927560 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:29.927473    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-86659d4467-rv8gb_e68bb827-8a52-4b49-b3b3-55c6c355d258/maas-api/0.log"
Jul 30 03:25:30.038414 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:30.038384    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29756325-dbq98_4ede0bf8-35cf-4b68-aa08-c9db0df578e3/cleanup/0.log"
Jul 30 03:25:30.142091 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:30.142060    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29756340-42grx_6a8e67e6-6266-4420-82b7-4f70e0678a9f/cleanup/0.log"
Jul 30 03:25:30.353420 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:30.353390    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-jkfmb_e03462aa-5f71-492f-92f4-7971c5ead0a8/postgres/0.log"
Jul 30 03:25:31.101029 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:31.100999    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx_c08a7ead-46f4-4139-bc12-4b18542dca53/util/0.log"
Jul 30 03:25:31.106685 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:31.106665    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx_c08a7ead-46f4-4139-bc12-4b18542dca53/pull/0.log"
Jul 30 03:25:31.112216 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:31.112196    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx_c08a7ead-46f4-4139-bc12-4b18542dca53/extract/0.log"
Jul 30 03:25:31.223160 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:31.223129    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r_ec4c7f41-225c-43a6-b720-6c566df46dae/util/0.log"
Jul 30 03:25:31.230260 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:31.230242    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r_ec4c7f41-225c-43a6-b720-6c566df46dae/pull/0.log"
Jul 30 03:25:31.236444 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:31.236424    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r_ec4c7f41-225c-43a6-b720-6c566df46dae/extract/0.log"
Jul 30 03:25:31.347638 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:31.347609    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v_d1953a83-132c-4c51-ab36-236e674af393/util/0.log"
Jul 30 03:25:31.354131 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:31.354066    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v_d1953a83-132c-4c51-ab36-236e674af393/pull/0.log"
Jul 30 03:25:31.360039 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:31.360020    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v_d1953a83-132c-4c51-ab36-236e674af393/extract/0.log"
Jul 30 03:25:31.467736 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:31.467705    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5_7d93308e-5b79-4104-a16e-03a7677fed49/pull/0.log"
Jul 30 03:25:31.474153 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:31.474131    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5_7d93308e-5b79-4104-a16e-03a7677fed49/extract/0.log"
Jul 30 03:25:31.480267 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:31.480248    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5_7d93308e-5b79-4104-a16e-03a7677fed49/util/0.log"
Jul 30 03:25:31.709237 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:31.709113    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-b265s_8b2ce1d9-ce1b-49e6-91f8-92c11bcba18c/manager/0.log"
Jul 30 03:25:31.815024 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:31.814996    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-n9xwx_703264f4-d1c9-4228-8600-11a897120f05/manager/0.log"
Jul 30 03:25:31.919331 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:31.919310    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-zsg96_f8fe9862-0773-4e89-b2b3-d4bd37967bbb/kuadrant-console-plugin/0.log"
Jul 30 03:25:32.030465 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:32.030381    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-4xdbc_fb3ddd39-1487-46cf-902d-58950c899a5e/registry-server/0.log"
Jul 30 03:25:32.145438 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:32.145409    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-djc9v_33d3c02b-b591-4da0-9178-3cfb7eafa488/manager/0.log"
Jul 30 03:25:32.713187 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:32.713153    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl_0c3ad8b3-6927-4c20-a313-51510a35bf13/istio-proxy/0.log"
Jul 30 03:25:33.043040 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:33.043016    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b44668b45-jz27k_8fe6e745-14ba-4e8b-b8b0-72be885b4169/kube-auth-proxy/0.log"
Jul 30 03:25:33.160800 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:33.160771    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9vf5j_cd921d07-bc19-45b2-b724-43dce72b1d84/istio-proxy/1.log"
Jul 30 03:25:33.393111 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:33.393026    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-wg4rd_0c013301-ce62-4de8-87b6-4e9e6ed43dba/payload-processing/0.log"
Jul 30 03:25:33.941524 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:33.941496    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-bts5f_bc883c92-489d-422d-8ef2-5ba476ab0291/main/0.log"
Jul 30 03:25:39.982553 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.982520    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-trtdw/must-gather-5zk49"]
Jul 30 03:25:39.982933 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.982890    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba45532-1c80-4328-b6d2-c73ff607f539" containerName="istio-proxy"
Jul 30 03:25:39.982933 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.982901    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba45532-1c80-4328-b6d2-c73ff607f539" containerName="istio-proxy"
Jul 30 03:25:39.982933 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.982911    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="990d6ed8-feb1-455b-ba3a-a5243bbd63fc" containerName="maas-api"
Jul 30 03:25:39.982933 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.982916    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="990d6ed8-feb1-455b-ba3a-a5243bbd63fc" containerName="maas-api"
Jul 30 03:25:39.982933 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.982923    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02bb64dc-de3e-4e34-b139-d6510f2eb893" containerName="istio-proxy"
Jul 30 03:25:39.982933 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.982928    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="02bb64dc-de3e-4e34-b139-d6510f2eb893" containerName="istio-proxy"
Jul 30 03:25:39.982933 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.982935    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3b9928-470e-40d0-a120-8041725bf0e3" containerName="istio-proxy"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.982941    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3b9928-470e-40d0-a120-8041725bf0e3" containerName="istio-proxy"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.982948    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" containerName="istio-proxy"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.982953    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" containerName="istio-proxy"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.982959    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3b9928-470e-40d0-a120-8041725bf0e3" containerName="istio-proxy"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.982964    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3b9928-470e-40d0-a120-8041725bf0e3" containerName="istio-proxy"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.982970    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02bb64dc-de3e-4e34-b139-d6510f2eb893" containerName="istio-proxy"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.982975    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="02bb64dc-de3e-4e34-b139-d6510f2eb893" containerName="istio-proxy"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.982984    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba45532-1c80-4328-b6d2-c73ff607f539" containerName="istio-proxy"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.982991    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba45532-1c80-4328-b6d2-c73ff607f539" containerName="istio-proxy"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.982997    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d7b8f82-f331-4697-be50-4f724fd68bd0" containerName="istio-proxy"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983001    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d7b8f82-f331-4697-be50-4f724fd68bd0" containerName="istio-proxy"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983014    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4fb5b4d-5973-4860-bbf3-854e36472540" containerName="maas-api"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983018    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4fb5b4d-5973-4860-bbf3-854e36472540" containerName="maas-api"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983024    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9775762d-34b7-4161-986a-d462a991aff0" containerName="maas-api"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983029    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9775762d-34b7-4161-986a-d462a991aff0" containerName="maas-api"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983079    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="02bb64dc-de3e-4e34-b139-d6510f2eb893" containerName="istio-proxy"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983086    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="02bb64dc-de3e-4e34-b139-d6510f2eb893" containerName="istio-proxy"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983093    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba45532-1c80-4328-b6d2-c73ff607f539" containerName="istio-proxy"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983097    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c3b9928-470e-40d0-a120-8041725bf0e3" containerName="istio-proxy"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983104    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4fb5b4d-5973-4860-bbf3-854e36472540" containerName="maas-api"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983110    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d7b8f82-f331-4697-be50-4f724fd68bd0" containerName="istio-proxy"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983115    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d7b8f82-f331-4697-be50-4f724fd68bd0" containerName="istio-proxy"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983120    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9775762d-34b7-4161-986a-d462a991aff0" containerName="maas-api"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983125    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c3b9928-470e-40d0-a120-8041725bf0e3" containerName="istio-proxy"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983131    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" containerName="istio-proxy"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983137    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="990d6ed8-feb1-455b-ba3a-a5243bbd63fc" containerName="maas-api"
Jul 30 03:25:39.983158 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983142    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" containerName="istio-proxy"
Jul 30 03:25:39.983977 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983222    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" containerName="istio-proxy"
Jul 30 03:25:39.983977 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983228    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a74a1dac-06b5-4ca7-b2a3-a624549d6f1d" containerName="istio-proxy"
Jul 30 03:25:39.983977 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983237    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d7b8f82-f331-4697-be50-4f724fd68bd0" containerName="istio-proxy"
Jul 30 03:25:39.983977 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983241    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d7b8f82-f331-4697-be50-4f724fd68bd0" containerName="istio-proxy"
Jul 30 03:25:39.983977 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.983291    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba45532-1c80-4328-b6d2-c73ff607f539" containerName="istio-proxy"
Jul 30 03:25:39.987716 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.987696    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-trtdw/must-gather-5zk49"
Jul 30 03:25:39.990273 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.990258    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-trtdw\"/\"kube-root-ca.crt\""
Jul 30 03:25:39.991472 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.991455    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-trtdw\"/\"openshift-service-ca.crt\""
Jul 30 03:25:39.991533 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.991496    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-trtdw\"/\"default-dockercfg-6rs4c\""
Jul 30 03:25:39.996065 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:39.996045    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-trtdw/must-gather-5zk49"]
Jul 30 03:25:40.128603 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:40.128568    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/421071c1-6552-40dc-ae16-ba7f847a0746-must-gather-output\") pod \"must-gather-5zk49\" (UID: \"421071c1-6552-40dc-ae16-ba7f847a0746\") " pod="openshift-must-gather-trtdw/must-gather-5zk49"
Jul 30 03:25:40.128603 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:40.128601    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q2zx\" (UniqueName: \"kubernetes.io/projected/421071c1-6552-40dc-ae16-ba7f847a0746-kube-api-access-5q2zx\") pod \"must-gather-5zk49\" (UID: \"421071c1-6552-40dc-ae16-ba7f847a0746\") " pod="openshift-must-gather-trtdw/must-gather-5zk49"
Jul 30 03:25:40.229472 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:40.229432    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/421071c1-6552-40dc-ae16-ba7f847a0746-must-gather-output\") pod \"must-gather-5zk49\" (UID: \"421071c1-6552-40dc-ae16-ba7f847a0746\") " pod="openshift-must-gather-trtdw/must-gather-5zk49"
Jul 30 03:25:40.229472 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:40.229472    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q2zx\" (UniqueName: \"kubernetes.io/projected/421071c1-6552-40dc-ae16-ba7f847a0746-kube-api-access-5q2zx\") pod \"must-gather-5zk49\" (UID: \"421071c1-6552-40dc-ae16-ba7f847a0746\") " pod="openshift-must-gather-trtdw/must-gather-5zk49"
Jul 30 03:25:40.229796 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:40.229772    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/421071c1-6552-40dc-ae16-ba7f847a0746-must-gather-output\") pod \"must-gather-5zk49\" (UID: \"421071c1-6552-40dc-ae16-ba7f847a0746\") " pod="openshift-must-gather-trtdw/must-gather-5zk49"
Jul 30 03:25:40.238257 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:40.238201    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q2zx\" (UniqueName: \"kubernetes.io/projected/421071c1-6552-40dc-ae16-ba7f847a0746-kube-api-access-5q2zx\") pod \"must-gather-5zk49\" (UID: \"421071c1-6552-40dc-ae16-ba7f847a0746\") " pod="openshift-must-gather-trtdw/must-gather-5zk49"
Jul 30 03:25:40.298012 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:40.297975    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-trtdw/must-gather-5zk49"
Jul 30 03:25:40.433823 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:40.433796    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-trtdw/must-gather-5zk49"]
Jul 30 03:25:40.435351 ip-10-0-132-89 kubenswrapper[2565]: W0730 03:25:40.435324    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod421071c1_6552_40dc_ae16_ba7f847a0746.slice/crio-a3587b56649f0418bf72f0854ce8ecb2a2dcdbb708caf56d5954679a2b0a1eec WatchSource:0}: Error finding container a3587b56649f0418bf72f0854ce8ecb2a2dcdbb708caf56d5954679a2b0a1eec: Status 404 returned error can't find the container with id a3587b56649f0418bf72f0854ce8ecb2a2dcdbb708caf56d5954679a2b0a1eec
Jul 30 03:25:40.437624 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:40.437607    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 03:25:40.546692 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:40.546659    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trtdw/must-gather-5zk49" event={"ID":"421071c1-6552-40dc-ae16-ba7f847a0746","Type":"ContainerStarted","Data":"a3587b56649f0418bf72f0854ce8ecb2a2dcdbb708caf56d5954679a2b0a1eec"}
Jul 30 03:25:41.555880 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:41.555838    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trtdw/must-gather-5zk49" event={"ID":"421071c1-6552-40dc-ae16-ba7f847a0746","Type":"ContainerStarted","Data":"76d5e5421247262c3110594fe05e4bf4f4615f0bc7765ed9bf5d409721f30090"}
Jul 30 03:25:41.556286 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:41.555897    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trtdw/must-gather-5zk49" event={"ID":"421071c1-6552-40dc-ae16-ba7f847a0746","Type":"ContainerStarted","Data":"4bc949c06f0a87d378bc0d33ce9c5928d4189254b73edbf59a9cdd348ac83571"}
Jul 30 03:25:41.572893 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:41.572829    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-trtdw/must-gather-5zk49" podStartSLOduration=1.773921195 podStartE2EDuration="2.572810641s" totalImagesPullingTime="798.889446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:25:41.571220441 +0000 UTC m=+3577.218003847" watchObservedRunningTime="2026-07-30 03:25:41.572810641 +0000 UTC m=+3577.219594045"
Jul 30 03:25:42.867570 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:42.867541    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fsv7s_30a8139c-c65b-45ea-aa17-645a0e4c634a/global-pull-secret-syncer/0.log"
Jul 30 03:25:42.956189 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:42.956134    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b5trq_a7211fce-5509-4ca2-8fb1-33a73122baec/konnectivity-agent/0.log"
Jul 30 03:25:43.031774 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:43.031745    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-89.ec2.internal_2615c8560d8f795d489b67609c7125ee/haproxy/0.log"
Jul 30 03:25:46.808196 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:46.808151    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx_c08a7ead-46f4-4139-bc12-4b18542dca53/extract/0.log"
Jul 30 03:25:46.830661 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:46.830617    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx_c08a7ead-46f4-4139-bc12-4b18542dca53/util/0.log"
Jul 30 03:25:46.853239 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:46.853211    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596hldx_c08a7ead-46f4-4139-bc12-4b18542dca53/pull/0.log"
Jul 30 03:25:46.882409 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:46.882341    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r_ec4c7f41-225c-43a6-b720-6c566df46dae/extract/0.log"
Jul 30 03:25:46.903904 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:46.903879    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r_ec4c7f41-225c-43a6-b720-6c566df46dae/util/0.log"
Jul 30 03:25:46.926089 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:46.926062    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04d84r_ec4c7f41-225c-43a6-b720-6c566df46dae/pull/0.log"
Jul 30 03:25:46.971751 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:46.971646    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v_d1953a83-132c-4c51-ab36-236e674af393/extract/0.log"
Jul 30 03:25:47.009884 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:47.009856    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v_d1953a83-132c-4c51-ab36-236e674af393/util/0.log"
Jul 30 03:25:47.059475 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:47.059373    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7368x9v_d1953a83-132c-4c51-ab36-236e674af393/pull/0.log"
Jul 30 03:25:47.089307 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:47.089273    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5_7d93308e-5b79-4104-a16e-03a7677fed49/extract/0.log"
Jul 30 03:25:47.115538 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:47.115511    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5_7d93308e-5b79-4104-a16e-03a7677fed49/util/0.log"
Jul 30 03:25:47.143391 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:47.143330    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dl2b5_7d93308e-5b79-4104-a16e-03a7677fed49/pull/0.log"
Jul 30 03:25:47.528754 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:47.528587    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-b265s_8b2ce1d9-ce1b-49e6-91f8-92c11bcba18c/manager/0.log"
Jul 30 03:25:47.563006 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:47.562963    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-n9xwx_703264f4-d1c9-4228-8600-11a897120f05/manager/0.log"
Jul 30 03:25:47.596531 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:47.596474    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-zsg96_f8fe9862-0773-4e89-b2b3-d4bd37967bbb/kuadrant-console-plugin/0.log"
Jul 30 03:25:47.661885 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:47.661830    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-4xdbc_fb3ddd39-1487-46cf-902d-58950c899a5e/registry-server/0.log"
Jul 30 03:25:47.741127 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:47.741056    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-djc9v_33d3c02b-b591-4da0-9178-3cfb7eafa488/manager/0.log"
Jul 30 03:25:49.498031 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:49.497993    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_13da4179-ab3f-40d0-b08b-0978fce78314/alertmanager/0.log"
Jul 30 03:25:49.518625 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:49.518593    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_13da4179-ab3f-40d0-b08b-0978fce78314/config-reloader/0.log"
Jul 30 03:25:49.543064 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:49.543015    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_13da4179-ab3f-40d0-b08b-0978fce78314/kube-rbac-proxy-web/0.log"
Jul 30 03:25:49.566289 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:49.566219    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_13da4179-ab3f-40d0-b08b-0978fce78314/kube-rbac-proxy/0.log"
Jul 30 03:25:49.586753 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:49.586718    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_13da4179-ab3f-40d0-b08b-0978fce78314/kube-rbac-proxy-metric/0.log"
Jul 30 03:25:49.605582 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:49.605498    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_13da4179-ab3f-40d0-b08b-0978fce78314/prom-label-proxy/0.log"
Jul 30 03:25:49.625446 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:49.625421    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_13da4179-ab3f-40d0-b08b-0978fce78314/init-config-reloader/0.log"
Jul 30 03:25:49.703902 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:49.703874    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-xn726_ced05f60-22f9-4ae7-a971-d530833ac4ad/kube-state-metrics/0.log"
Jul 30 03:25:49.728090 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:49.728054    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-xn726_ced05f60-22f9-4ae7-a971-d530833ac4ad/kube-rbac-proxy-main/0.log"
Jul 30 03:25:49.753390 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:49.753281    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-xn726_ced05f60-22f9-4ae7-a971-d530833ac4ad/kube-rbac-proxy-self/0.log"
Jul 30 03:25:49.946423 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:49.946341    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dn7g6_4ce518f3-8390-4e94-b88a-cdbe86cc1d7b/node-exporter/0.log"
Jul 30 03:25:49.985673 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:49.985648    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dn7g6_4ce518f3-8390-4e94-b88a-cdbe86cc1d7b/kube-rbac-proxy/0.log"
Jul 30 03:25:50.018258 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:50.018119    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dn7g6_4ce518f3-8390-4e94-b88a-cdbe86cc1d7b/init-textfile/0.log"
Jul 30 03:25:50.434227 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:50.434186    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-kdwrm_65ed1700-1134-46d6-85f1-2619f1d9986f/prometheus-operator-admission-webhook/0.log"
Jul 30 03:25:53.145997 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:53.145966    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-tjx92_e16dc4ef-22ab-442d-b030-b5903e9c92f1/download-server/0.log"
Jul 30 03:25:53.501321 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:53.501229    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"]
Jul 30 03:25:53.510096 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:53.510062    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"
Jul 30 03:25:53.513193 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:53.513126    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"]
Jul 30 03:25:53.581909 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:53.581871    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd766ad1-5093-454d-acfa-9794d7f258bb-lib-modules\") pod \"perf-node-gather-daemonset-bdzbs\" (UID: \"fd766ad1-5093-454d-acfa-9794d7f258bb\") " pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"
Jul 30 03:25:53.582121 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:53.581939    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fd766ad1-5093-454d-acfa-9794d7f258bb-podres\") pod \"perf-node-gather-daemonset-bdzbs\" (UID: \"fd766ad1-5093-454d-acfa-9794d7f258bb\") " pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"
Jul 30 03:25:53.582121 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:53.581970    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd766ad1-5093-454d-acfa-9794d7f258bb-sys\") pod \"perf-node-gather-daemonset-bdzbs\" (UID: \"fd766ad1-5093-454d-acfa-9794d7f258bb\") " pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"
Jul 30 03:25:53.582121 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:53.581999    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fd766ad1-5093-454d-acfa-9794d7f258bb-proc\") pod \"perf-node-gather-daemonset-bdzbs\" (UID: \"fd766ad1-5093-454d-acfa-9794d7f258bb\") " pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"
Jul 30 03:25:53.582121 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:53.582052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb6vd\" (UniqueName: \"kubernetes.io/projected/fd766ad1-5093-454d-acfa-9794d7f258bb-kube-api-access-tb6vd\") pod \"perf-node-gather-daemonset-bdzbs\" (UID: \"fd766ad1-5093-454d-acfa-9794d7f258bb\") " pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"
Jul 30 03:25:53.683526 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:53.683480    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fd766ad1-5093-454d-acfa-9794d7f258bb-podres\") pod \"perf-node-gather-daemonset-bdzbs\" (UID: \"fd766ad1-5093-454d-acfa-9794d7f258bb\") " pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"
Jul 30 03:25:53.683727 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:53.683537    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd766ad1-5093-454d-acfa-9794d7f258bb-sys\") pod \"perf-node-gather-daemonset-bdzbs\" (UID: \"fd766ad1-5093-454d-acfa-9794d7f258bb\") " pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"
Jul 30 03:25:53.683727 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:53.683572    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fd766ad1-5093-454d-acfa-9794d7f258bb-proc\") pod \"perf-node-gather-daemonset-bdzbs\" (UID: \"fd766ad1-5093-454d-acfa-9794d7f258bb\") " pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"
Jul 30 03:25:53.683727 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:53.683632    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb6vd\" (UniqueName: \"kubernetes.io/projected/fd766ad1-5093-454d-acfa-9794d7f258bb-kube-api-access-tb6vd\") pod \"perf-node-gather-daemonset-bdzbs\" (UID: \"fd766ad1-5093-454d-acfa-9794d7f258bb\") " pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"
Jul 30 03:25:53.683727 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:53.683720    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd766ad1-5093-454d-acfa-9794d7f258bb-lib-modules\") pod \"perf-node-gather-daemonset-bdzbs\" (UID: \"fd766ad1-5093-454d-acfa-9794d7f258bb\") " pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"
Jul 30 03:25:53.683952 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:53.683893    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd766ad1-5093-454d-acfa-9794d7f258bb-lib-modules\") pod \"perf-node-gather-daemonset-bdzbs\" (UID: \"fd766ad1-5093-454d-acfa-9794d7f258bb\") " pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"
Jul 30 03:25:53.684013 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:53.683974    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fd766ad1-5093-454d-acfa-9794d7f258bb-podres\") pod \"perf-node-gather-daemonset-bdzbs\" (UID: \"fd766ad1-5093-454d-acfa-9794d7f258bb\") " pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"
Jul 30 03:25:53.684069 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:53.684020    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd766ad1-5093-454d-acfa-9794d7f258bb-sys\") pod \"perf-node-gather-daemonset-bdzbs\" (UID: \"fd766ad1-5093-454d-acfa-9794d7f258bb\") " pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"
Jul 30 03:25:53.684125 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:53.684066    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fd766ad1-5093-454d-acfa-9794d7f258bb-proc\") pod \"perf-node-gather-daemonset-bdzbs\" (UID: \"fd766ad1-5093-454d-acfa-9794d7f258bb\") " pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"
Jul 30 03:25:53.693211 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:53.693178    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb6vd\" (UniqueName: \"kubernetes.io/projected/fd766ad1-5093-454d-acfa-9794d7f258bb-kube-api-access-tb6vd\") pod \"perf-node-gather-daemonset-bdzbs\" (UID: \"fd766ad1-5093-454d-acfa-9794d7f258bb\") " pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"
Jul 30 03:25:53.823075 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:53.823023    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"
Jul 30 03:25:54.023452 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:54.023424    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"]
Jul 30 03:25:54.667043 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:54.666993    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs" event={"ID":"fd766ad1-5093-454d-acfa-9794d7f258bb","Type":"ContainerStarted","Data":"db3087cabb199be2d7b7b934e0853e15ab8c9f7573e8342067ad736c65d9d6ff"}
Jul 30 03:25:54.667502 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:54.667067    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs" event={"ID":"fd766ad1-5093-454d-acfa-9794d7f258bb","Type":"ContainerStarted","Data":"3397e64fbb53da9684d6753d903f41b5769e725fd44ae0b43fddeb0f16bf805d"}
Jul 30 03:25:54.667502 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:54.667102    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"
Jul 30 03:25:54.686137 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:54.686069    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs" podStartSLOduration=1.6860480230000001 podStartE2EDuration="1.686048023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:25:54.684635313 +0000 UTC m=+3590.331418718" watchObservedRunningTime="2026-07-30 03:25:54.686048023 +0000 UTC m=+3590.332831428"
Jul 30 03:25:54.970490 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:54.970352    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7rhjt_6641518e-6c07-41e1-92b0-470be0e94ddc/dns/0.log"
Jul 30 03:25:54.989450 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:54.989416    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7rhjt_6641518e-6c07-41e1-92b0-470be0e94ddc/kube-rbac-proxy/0.log"
Jul 30 03:25:55.070996 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:55.070971    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bzccb_648853ed-378d-4c9d-82e8-5cbb958399db/dns-node-resolver/0.log"
Jul 30 03:25:55.617916 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:55.617866    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6d95b6b5f4-ccppl_ad67c765-01ee-460c-b0ce-9da7d9c8a14e/registry/0.log"
Jul 30 03:25:55.665270 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:55.665120    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bspvc_1f1a6683-1bcf-48ce-a20b-525984e9a23a/node-ca/0.log"
Jul 30 03:25:56.604627 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:56.604593    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl_0c3ad8b3-6927-4c20-a313-51510a35bf13/istio-proxy/0.log"
Jul 30 03:25:56.791145 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:56.791117    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b44668b45-jz27k_8fe6e745-14ba-4e8b-b8b0-72be885b4169/kube-auth-proxy/0.log"
Jul 30 03:25:56.815691 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:56.815659    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9vf5j_cd921d07-bc19-45b2-b724-43dce72b1d84/istio-proxy/0.log"
Jul 30 03:25:56.851658 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:56.851630    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9vf5j_cd921d07-bc19-45b2-b724-43dce72b1d84/istio-proxy/1.log"
Jul 30 03:25:56.918220 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:56.918115    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-wg4rd_0c013301-ce62-4de8-87b6-4e9e6ed43dba/payload-processing/0.log"
Jul 30 03:25:57.782567 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:57.782537    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8gs9b_d4dfc729-47bc-41b7-b64d-0c1b45010108/serve-healthcheck-canary/0.log"
Jul 30 03:25:58.314139 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:58.314084    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9z4l6_09a07cf3-a768-4fcd-ade4-85783f4bf6d6/kube-rbac-proxy/0.log"
Jul 30 03:25:58.337287 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:58.337260    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9z4l6_09a07cf3-a768-4fcd-ade4-85783f4bf6d6/exporter/0.log"
Jul 30 03:25:58.357584 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:25:58.357559    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9z4l6_09a07cf3-a768-4fcd-ade4-85783f4bf6d6/extractor/0.log"
Jul 30 03:26:00.474357 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:00.474329    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5kntq_cf4f1044-78f4-4086-aac0-406665379140/manager/0.log"
Jul 30 03:26:00.685503 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:00.685454    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-trtdw/perf-node-gather-daemonset-bdzbs"
Jul 30 03:26:00.762805 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:00.762722    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-x28m9_6e9e9962-00bd-4256-a254-276c940b6d80/manager/0.log"
Jul 30 03:26:00.946189 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:00.945870    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6899b54cb8-4k7tc_4b1585e2-ab06-4fc4-ad31-c3cf9912ea2b/manager/0.log"
Jul 30 03:26:01.246409 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:01.246369    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-684d7c5d99-sc8pn_93cde41a-963f-4f29-814d-4c909d54efa3/manager/0.log"
Jul 30 03:26:01.267332 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:01.267300    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-684d7c5d99-sc8pn_93cde41a-963f-4f29-814d-4c909d54efa3/copy-manifests/0.log"
Jul 30 03:26:02.641279 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:02.641154    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-5blr2_f46c6b01-8d08-4631-bfdb-523159676aad/openshift-lws-operator/0.log"
Jul 30 03:26:03.883854 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:03.883814    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl_0c3ad8b3-6927-4c20-a313-51510a35bf13/istio-proxy/0.log"
Jul 30 03:26:04.063072 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:04.063032    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b44668b45-jz27k_8fe6e745-14ba-4e8b-b8b0-72be885b4169/kube-auth-proxy/0.log"
Jul 30 03:26:04.080523 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:04.080489    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9vf5j_cd921d07-bc19-45b2-b724-43dce72b1d84/istio-proxy/0.log"
Jul 30 03:26:04.117634 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:04.117604    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9vf5j_cd921d07-bc19-45b2-b724-43dce72b1d84/istio-proxy/1.log"
Jul 30 03:26:04.161532 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:04.161478    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-wg4rd_0c013301-ce62-4de8-87b6-4e9e6ed43dba/payload-processing/0.log"
Jul 30 03:26:05.481127 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:05.481072    2565 scope.go:117] "RemoveContainer" containerID="4d5e779f025c182423c5dc32d071b760448815417c605adbff4c4bf783b4dd9a"
Jul 30 03:26:05.702965 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:05.702933    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-7mtwl_0c3ad8b3-6927-4c20-a313-51510a35bf13/istio-proxy/0.log"
Jul 30 03:26:05.877698 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:05.877664    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b44668b45-jz27k_8fe6e745-14ba-4e8b-b8b0-72be885b4169/kube-auth-proxy/0.log"
Jul 30 03:26:05.901307 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:05.901278    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9vf5j_cd921d07-bc19-45b2-b724-43dce72b1d84/istio-proxy/0.log"
Jul 30 03:26:05.940393 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:05.940353    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9vf5j_cd921d07-bc19-45b2-b724-43dce72b1d84/istio-proxy/1.log"
Jul 30 03:26:05.985525 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:05.985494    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-wg4rd_0c013301-ce62-4de8-87b6-4e9e6ed43dba/payload-processing/0.log"
Jul 30 03:26:09.279125 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:09.279099    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-77n5d_fc2726a6-0c55-49f0-823e-58839b890c37/kube-multus/0.log"
Jul 30 03:26:09.330140 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:09.330112    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-278lc_b34ac4fa-e29d-4585-80fe-928a134067aa/kube-multus-additional-cni-plugins/0.log"
Jul 30 03:26:09.348988 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:09.348961    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-278lc_b34ac4fa-e29d-4585-80fe-928a134067aa/egress-router-binary-copy/0.log"
Jul 30 03:26:09.375587 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:09.375555    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-278lc_b34ac4fa-e29d-4585-80fe-928a134067aa/cni-plugins/0.log"
Jul 30 03:26:09.393600 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:09.393570    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-278lc_b34ac4fa-e29d-4585-80fe-928a134067aa/bond-cni-plugin/0.log"
Jul 30 03:26:09.411979 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:09.411955    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-278lc_b34ac4fa-e29d-4585-80fe-928a134067aa/routeoverride-cni/0.log"
Jul 30 03:26:09.430389 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:09.430360    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-278lc_b34ac4fa-e29d-4585-80fe-928a134067aa/whereabouts-cni-bincopy/0.log"
Jul 30 03:26:09.448710 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:09.448681    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-278lc_b34ac4fa-e29d-4585-80fe-928a134067aa/whereabouts-cni/0.log"
Jul 30 03:26:09.851089 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:09.851051    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wphj8_ec397db7-4d16-4a1d-9ace-fb4de6e17039/network-metrics-daemon/0.log"
Jul 30 03:26:09.865800 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:09.865771    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wphj8_ec397db7-4d16-4a1d-9ace-fb4de6e17039/kube-rbac-proxy/0.log"
Jul 30 03:26:10.707752 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:10.707714    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rrn8_3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b/ovn-controller/0.log"
Jul 30 03:26:10.725305 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:10.725280    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rrn8_3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b/ovn-acl-logging/0.log"
Jul 30 03:26:10.750804 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:10.750772    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rrn8_3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b/kube-rbac-proxy-node/0.log"
Jul 30 03:26:10.771673 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:10.771615    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rrn8_3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b/kube-rbac-proxy-ovn-metrics/0.log"
Jul 30 03:26:10.788030 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:10.787996    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rrn8_3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b/northd/0.log"
Jul 30 03:26:10.805646 ip-10-0-132-89 kubenswrapper[2565]: I0730 03:26:10.805618    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rrn8_3b5bcf49-0c5c-4a7b-83ba-7bc1167f730b/nbdb/0.log"