Sep 11 14:40:20.110153 ip-10-0-133-126 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 14:40:20.446435 ip-10-0-133-126 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 14:40:20.446435 ip-10-0-133-126 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 14:40:20.446435 ip-10-0-133-126 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 14:40:20.446435 ip-10-0-133-126 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 14:40:20.446435 ip-10-0-133-126 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 14:40:20.447865 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.447772    2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 14:40:20.453381 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453363    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 14:40:20.453381 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453380    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 14:40:20.453381 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453384    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 14:40:20.453485 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453387    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 14:40:20.453485 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453390    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 14:40:20.453485 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453393    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 14:40:20.453485 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453396    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 14:40:20.453485 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453401    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 14:40:20.453485 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453405    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 14:40:20.453485 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453408    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 14:40:20.453485 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453411    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 14:40:20.453485 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453414    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 14:40:20.453485 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453417    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 14:40:20.453485 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453420    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 14:40:20.453485 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453423    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 14:40:20.453485 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453426    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 14:40:20.453485 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453428    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 14:40:20.453485 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453431    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 14:40:20.453485 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453433    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 14:40:20.453485 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453436    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 14:40:20.453485 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453442    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 14:40:20.453485 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453445    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 14:40:20.453944 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453447    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 14:40:20.453944 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453450    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 14:40:20.453944 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453452    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 14:40:20.453944 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453455    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 14:40:20.453944 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453458    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 14:40:20.453944 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453460    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 14:40:20.453944 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453462    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 14:40:20.453944 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453465    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 14:40:20.453944 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453467    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 14:40:20.453944 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453470    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 14:40:20.453944 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453473    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 14:40:20.453944 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453475    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 14:40:20.453944 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453478    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 14:40:20.453944 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453480    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 14:40:20.453944 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453484    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 14:40:20.453944 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453487    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 14:40:20.453944 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453490    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 14:40:20.453944 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453492    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 14:40:20.453944 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453495    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 14:40:20.454438 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453498    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 14:40:20.454438 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453501    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 14:40:20.454438 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453504    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 14:40:20.454438 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453507    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 14:40:20.454438 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453510    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 14:40:20.454438 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453513    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 14:40:20.454438 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453515    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 14:40:20.454438 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453518    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 14:40:20.454438 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453520    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 14:40:20.454438 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453523    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 14:40:20.454438 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453525    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 14:40:20.454438 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453528    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 14:40:20.454438 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453530    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 14:40:20.454438 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453533    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 14:40:20.454438 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453535    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 14:40:20.454438 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453538    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 14:40:20.454438 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453540    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 14:40:20.454438 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453543    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 14:40:20.454438 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453545    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 14:40:20.454917 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453548    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 14:40:20.454917 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453550    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 14:40:20.454917 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453553    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 14:40:20.454917 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453555    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 14:40:20.454917 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453559    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 14:40:20.454917 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453562    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 14:40:20.454917 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453564    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 14:40:20.454917 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453569    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 14:40:20.454917 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453572    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 14:40:20.454917 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453575    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 14:40:20.454917 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453577    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 14:40:20.454917 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453581    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 14:40:20.454917 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453584    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 14:40:20.454917 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453586    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 14:40:20.454917 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453591    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 14:40:20.454917 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453594    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 14:40:20.454917 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453597    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 14:40:20.454917 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453600    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 14:40:20.454917 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453602    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 14:40:20.454917 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453605    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 14:40:20.455414 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453607    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 14:40:20.455414 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453610    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 14:40:20.455414 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453612    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 14:40:20.455414 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453615    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 14:40:20.455414 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453619    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 14:40:20.455414 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453621    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 14:40:20.455414 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453624    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 14:40:20.455414 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453626    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 14:40:20.455414 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453629    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 14:40:20.455414 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453632    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 14:40:20.455414 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453635    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 14:40:20.455414 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453637    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 14:40:20.455414 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453640    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 14:40:20.455414 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453642    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 14:40:20.455414 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453644    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 14:40:20.455414 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453647    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 14:40:20.455414 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453651    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 14:40:20.455414 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453654    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 14:40:20.455414 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.453657    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 14:40:20.455414 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.454975    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 14:40:20.455903 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.454985    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 14:40:20.455903 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.454988    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 14:40:20.455903 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.454991    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 14:40:20.455903 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.454994    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 14:40:20.455903 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.454997    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 14:40:20.455903 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455000    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 14:40:20.455903 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455003    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 14:40:20.455903 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455006    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 14:40:20.455903 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455008    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 14:40:20.455903 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455011    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 14:40:20.455903 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455013    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 14:40:20.455903 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455016    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 14:40:20.455903 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455019    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 14:40:20.455903 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455022    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 14:40:20.455903 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455025    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 14:40:20.455903 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455028    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 14:40:20.455903 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455031    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 14:40:20.455903 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455033    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 14:40:20.455903 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455036    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 14:40:20.455903 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455038    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 14:40:20.456392 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455041    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 14:40:20.456392 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455043    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 14:40:20.456392 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455046    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 14:40:20.456392 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455049    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 14:40:20.456392 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455052    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 14:40:20.456392 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455056    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 14:40:20.456392 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455060    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 14:40:20.456392 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455062    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 14:40:20.456392 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455065    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 14:40:20.456392 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455067    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 14:40:20.456392 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455070    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 14:40:20.456392 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455073    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 14:40:20.456392 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455076    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 14:40:20.456392 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455079    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 14:40:20.456392 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455081    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 14:40:20.456392 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455084    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 14:40:20.456392 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455086    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 14:40:20.456392 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455089    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 14:40:20.456392 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455092    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 14:40:20.456392 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455095    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 14:40:20.456892 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455097    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 14:40:20.456892 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455100    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 14:40:20.456892 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455102    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 14:40:20.456892 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455105    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 14:40:20.456892 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455109    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 14:40:20.456892 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455113    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 14:40:20.456892 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455116    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 14:40:20.456892 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455119    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 14:40:20.456892 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455122    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 14:40:20.456892 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455125    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 14:40:20.456892 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455129    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 14:40:20.456892 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455132    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 14:40:20.456892 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455135    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 14:40:20.456892 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455138    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 14:40:20.456892 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455141    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 14:40:20.456892 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455144    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 14:40:20.456892 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455147    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 14:40:20.456892 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455150    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 14:40:20.456892 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455152    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 14:40:20.457360 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455155    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 14:40:20.457360 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455158    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 14:40:20.457360 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455161    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 14:40:20.457360 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455164    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 14:40:20.457360 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455167    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 14:40:20.457360 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455169    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 14:40:20.457360 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455173    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 14:40:20.457360 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455176    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 14:40:20.457360 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455178    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 14:40:20.457360 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455181    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 14:40:20.457360 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455184    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 14:40:20.457360 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455187    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 14:40:20.457360 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455189    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 14:40:20.457360 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455192    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 14:40:20.457360 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455194    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 14:40:20.457360 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455197    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 14:40:20.457360 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455199    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 14:40:20.457360 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455202    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 14:40:20.457360 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455204    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 14:40:20.457360 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455207    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 14:40:20.457857 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455209    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 14:40:20.457857 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455212    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 14:40:20.457857 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455214    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 14:40:20.457857 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455217    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 14:40:20.457857 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455219    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 14:40:20.457857 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455222    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 14:40:20.457857 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455224    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 14:40:20.457857 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455227    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 14:40:20.457857 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455229    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 14:40:20.457857 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455232    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 14:40:20.457857 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455235    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 14:40:20.457857 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455237    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 14:40:20.457857 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455239    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 14:40:20.457857 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455242    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 14:40:20.457857 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455244    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 14:40:20.457857 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455247    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 14:40:20.457857 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455256    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 14:40:20.457857 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455259    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 14:40:20.457857 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455262    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 14:40:20.457857 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455638    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 14:40:20.458357 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455642    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 14:40:20.458357 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455647    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 14:40:20.458357 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455650    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 14:40:20.458357 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455653    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 14:40:20.458357 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455656    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 14:40:20.458357 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455658    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 14:40:20.458357 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455661    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 14:40:20.458357 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455664    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 14:40:20.458357 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455668    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 14:40:20.458357 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455672    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 14:40:20.458357 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455675    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 14:40:20.458357 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455678    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 14:40:20.458357 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455681    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 14:40:20.458357 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455684    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 14:40:20.458357 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455686    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 14:40:20.458357 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455689    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 14:40:20.458357 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455692    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 14:40:20.458357 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455694    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 14:40:20.458357 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455697    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 14:40:20.458914 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455699    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 14:40:20.458914 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455702    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 14:40:20.458914 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455705    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 14:40:20.458914 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455707    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 14:40:20.458914 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455709    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 14:40:20.458914 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455712    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 14:40:20.458914 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455715    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 14:40:20.458914 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455717    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 14:40:20.458914 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455720    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 14:40:20.458914 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455723    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 14:40:20.458914 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455725    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 14:40:20.458914 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455728    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 14:40:20.458914 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455730    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 14:40:20.458914 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455733    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 14:40:20.458914 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455737    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 14:40:20.458914 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455740    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 14:40:20.458914 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455742    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 14:40:20.458914 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455745    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 14:40:20.458914 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455748    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 14:40:20.458914 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455751    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 14:40:20.459418 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455753    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 14:40:20.459418 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455756    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 14:40:20.459418 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455758    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 14:40:20.459418 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455761    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 14:40:20.459418 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455764    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 14:40:20.459418 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455766    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 14:40:20.459418 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455769    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 14:40:20.459418 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455771    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 14:40:20.459418 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455774    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 14:40:20.459418 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455776    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 14:40:20.459418 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455779    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 14:40:20.459418 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455781    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 14:40:20.459418 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455784    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 14:40:20.459418 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455786    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 14:40:20.459418 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455789    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 14:40:20.459418 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455793    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 14:40:20.459418 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455797    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 14:40:20.459418 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455799    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 14:40:20.459418 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455802    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 14:40:20.459418 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455805    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 14:40:20.459951 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455809    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 14:40:20.459951 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455825    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 14:40:20.459951 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455828    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 14:40:20.459951 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455832    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 14:40:20.459951 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455835    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 14:40:20.459951 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455838    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 14:40:20.459951 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455842    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 14:40:20.459951 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455844    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 14:40:20.459951 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455847    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 14:40:20.459951 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455850    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 14:40:20.459951 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455853    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 14:40:20.459951 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455856    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 14:40:20.459951 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455858    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 14:40:20.459951 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455861    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 14:40:20.459951 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455864    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 14:40:20.459951 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455866    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 14:40:20.459951 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455870    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 14:40:20.459951 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455872    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 14:40:20.459951 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455875    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 14:40:20.459951 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455877    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 14:40:20.460440 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455881    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 14:40:20.460440 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455883    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 14:40:20.460440 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455886    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 14:40:20.460440 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455888    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 14:40:20.460440 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455891    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 14:40:20.460440 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455893    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 14:40:20.460440 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455896    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 14:40:20.460440 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455898    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 14:40:20.460440 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455902    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 14:40:20.460440 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455905    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 14:40:20.460440 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455908    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 14:40:20.460440 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455910    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 14:40:20.460440 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455916    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 14:40:20.460440 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455919    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 14:40:20.460440 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455921    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 14:40:20.460440 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455924    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 14:40:20.460440 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455926    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 14:40:20.460440 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455929    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 14:40:20.460440 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.455932    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 14:40:20.460440 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456004    2568 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456036    2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456043    2568 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456048    2568 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456053    2568 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456056    2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456060    2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456065    2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456068    2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456071    2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456074    2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456078    2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456081    2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456084    2568 flags.go:64] FLAG: --cgroup-root=""
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456087    2568 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456090    2568 flags.go:64] FLAG: --client-ca-file=""
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456093    2568 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456095    2568 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456100    2568 flags.go:64] FLAG: --cluster-domain=""
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456103    2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456106    2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456109    2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456112    2568 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456116    2568 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 14:40:20.460916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456119    2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456124    2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456127    2568 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456130    2568 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456133    2568 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456136    2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456139    2568 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456142    2568 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456146    2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456150    2568 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456152    2568 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456156    2568 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456159    2568 flags.go:64] FLAG: --enable-server="true"
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456162    2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456166    2568 flags.go:64] FLAG: --event-burst="100"
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456169    2568 flags.go:64] FLAG: --event-qps="50"
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456172    2568 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456175    2568 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456178    2568 flags.go:64] FLAG: --eviction-hard=""
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456181    2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456184    2568 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456187    2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456190    2568 flags.go:64] FLAG: --eviction-soft=""
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456193    2568 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456196    2568 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 14:40:20.461519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456199    2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456202    2568 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456205    2568 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456208    2568 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456211    2568 flags.go:64] FLAG: --feature-gates=""
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456214    2568 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456217    2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456221    2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456225    2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456228    2568 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456231    2568 flags.go:64] FLAG: --help="false"
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456234    2568 flags.go:64] FLAG: --hostname-override="ip-10-0-133-126.ec2.internal"
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456237    2568 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456240    2568 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456244    2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456247    2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456250    2568 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456253    2568 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456256    2568 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456259    2568 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456262    2568 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456265    2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456268    2568 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456271    2568 flags.go:64] FLAG: --kube-reserved=""
Sep 11 14:40:20.462144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456274    2568 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456277    2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456280    2568 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456283    2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456285    2568 flags.go:64] FLAG: --lock-file=""
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456288    2568 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456291    2568 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456294    2568 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456299    2568 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456302    2568 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456305    2568 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456307    2568 flags.go:64] FLAG: --logging-format="text"
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456310    2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456314    2568 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456316    2568 flags.go:64] FLAG: --manifest-url=""
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456319    2568 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456325    2568 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456328    2568 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456332    2568 flags.go:64] FLAG: --max-pods="110"
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456335    2568 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456338    2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456340    2568 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456343    2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456346    2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456349    2568 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 14:40:20.462732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456352    2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456359    2568 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456362    2568 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456365    2568 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456368    2568 flags.go:64] FLAG: --pod-cidr=""
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456372    2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456377    2568 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456380    2568 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456383    2568 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456386    2568 flags.go:64] FLAG: --port="10250"
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456389    2568 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456392    2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d337064bc2d28d8c"
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456395    2568 flags.go:64] FLAG: --qos-reserved=""
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456399    2568 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456402    2568 flags.go:64] FLAG: --register-node="true"
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456405    2568 flags.go:64] FLAG: --register-with-taints=""
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456408    2568 flags.go:64] FLAG: --registry-burst="10"
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456411    2568 flags.go:64] FLAG: --registry-qps="5"
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456414    2568 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456417    2568 flags.go:64] FLAG: --reserved-memory=""
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456420    2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456423    2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456426    2568 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456430    2568 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456433    2568 flags.go:64] FLAG: --runonce="false"
Sep 11 14:40:20.463377 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456436    2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456439    2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456442    2568 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456445    2568 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456448    2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456453    2568 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456456    2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456459    2568 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456462    2568 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456465    2568 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456468    2568 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456471    2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456474    2568 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456477    2568 flags.go:64] FLAG: --system-cgroups=""
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456480    2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456485    2568 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456488    2568 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456491    2568 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456495    2568 flags.go:64] FLAG: --tls-min-version=""
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456498    2568 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456501    2568 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456504    2568 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456507    2568 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456510    2568 flags.go:64] FLAG: --v="2"
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456514    2568 flags.go:64] FLAG: --version="false"
Sep 11 14:40:20.464014 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456518    2568 flags.go:64] FLAG: --vmodule=""
Sep 11 14:40:20.464642 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456521    2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 14:40:20.464642 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.456524    2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 14:40:20.464642 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456639    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 14:40:20.464642 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456643    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 14:40:20.464642 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456647    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 14:40:20.464642 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456650    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 14:40:20.464642 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456653    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 14:40:20.464642 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456656    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 14:40:20.464642 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456659    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 14:40:20.464642 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456661    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 14:40:20.464642 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456664    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 14:40:20.464642 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456668    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 14:40:20.464642 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456671    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 14:40:20.464642 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456674    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 14:40:20.464642 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456677    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 14:40:20.464642 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456681    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 14:40:20.464642 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456685    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 14:40:20.464642 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456689    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 14:40:20.464642 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456693    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 14:40:20.464642 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456696    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 14:40:20.465148 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456702    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 14:40:20.465148 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456705    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 14:40:20.465148 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456709    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 14:40:20.465148 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456712    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 14:40:20.465148 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456715    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 14:40:20.465148 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456718    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 14:40:20.465148 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456721    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 14:40:20.465148 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456724    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 14:40:20.465148 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456727    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 14:40:20.465148 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456729    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 14:40:20.465148 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456732    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 14:40:20.465148 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456736    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 14:40:20.465148 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456740    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 14:40:20.465148 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456742    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 14:40:20.465148 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456745    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 14:40:20.465148 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456748    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 14:40:20.465148 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456753    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 14:40:20.465148 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456757    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 14:40:20.465148 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456761    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 14:40:20.465609 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456764    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 14:40:20.465609 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456767    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 14:40:20.465609 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456769    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 14:40:20.465609 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456772    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 14:40:20.465609 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456776    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 14:40:20.465609 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456778    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 14:40:20.465609 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456781    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 14:40:20.465609 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456784    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 14:40:20.465609 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456786    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 14:40:20.465609 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456789    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 14:40:20.465609 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456791    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 14:40:20.465609 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456794    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 14:40:20.465609 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456797    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 14:40:20.465609 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456799    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 14:40:20.465609 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456802    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 14:40:20.465609 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456805    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 14:40:20.465609 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456808    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 14:40:20.465609 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456810    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 14:40:20.465609 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456825    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 14:40:20.465609 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456828    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 14:40:20.466107 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456831    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 14:40:20.466107 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456834    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 14:40:20.466107 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456836    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 14:40:20.466107 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456839    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 14:40:20.466107 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456842    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 14:40:20.466107 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456844    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 14:40:20.466107 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456847    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 14:40:20.466107 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456850    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 14:40:20.466107 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456853    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 14:40:20.466107 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456857    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 14:40:20.466107 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456859    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 14:40:20.466107 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456862    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 14:40:20.466107 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456864    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 14:40:20.466107 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456867    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 14:40:20.466107 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456869    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 14:40:20.466107 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456872    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 14:40:20.466107 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456876    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 14:40:20.466107 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456879    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 14:40:20.466107 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456881    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 14:40:20.466107 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456884    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 14:40:20.466591 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456887    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 14:40:20.466591 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456890    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 14:40:20.466591 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456892    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 14:40:20.466591 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456895    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 14:40:20.466591 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456898    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 14:40:20.466591 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456901    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 14:40:20.466591 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456903    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 14:40:20.466591 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456907    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 14:40:20.466591 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456909    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 14:40:20.466591 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456912    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 14:40:20.466591 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456915    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 14:40:20.466591 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456917    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 14:40:20.466591 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456920    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 14:40:20.466591 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456922    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 14:40:20.466591 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456925    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 14:40:20.466591 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456927    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 14:40:20.466591 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456930    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 14:40:20.466591 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456932    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 14:40:20.466591 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456935    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 14:40:20.466591 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456938    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 14:40:20.467179 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456941    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 14:40:20.467179 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.456946    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 14:40:20.467179 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.457505    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 14:40:20.467179 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.464415    2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 14:40:20.467179 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.464435    2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 14:40:20.467179 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.465094    2568 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 14:40:20.467179 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.465102    2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 14:40:20.467179 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465156    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 14:40:20.467179 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465161    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 14:40:20.467179 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465165    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 14:40:20.467179 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465168    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 14:40:20.467179 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465171    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 14:40:20.467179 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465174    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 14:40:20.467179 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465177    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 14:40:20.467179 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465180    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 14:40:20.467692 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465182    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 14:40:20.467692 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465185    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 14:40:20.467692 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465188    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 14:40:20.467692 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465191    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 14:40:20.467692 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465194    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 14:40:20.467692 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465197    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 14:40:20.467692 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465199    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 14:40:20.467692 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465203    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 14:40:20.467692 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465205    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 14:40:20.467692 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465208    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 14:40:20.467692 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465212    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 14:40:20.467692 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465215    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 14:40:20.467692 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465218    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 14:40:20.467692 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465221    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 14:40:20.467692 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465224    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 14:40:20.467692 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465228    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 14:40:20.467692 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465231    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 14:40:20.467692 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465233    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 14:40:20.467692 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465236    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 14:40:20.468368 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465238    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 14:40:20.468368 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465241    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 14:40:20.468368 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465245    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 14:40:20.468368 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465248    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 14:40:20.468368 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465251    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 14:40:20.468368 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465254    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 14:40:20.468368 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465256    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 14:40:20.468368 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465259    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 14:40:20.468368 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465262    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 14:40:20.468368 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465264    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 14:40:20.468368 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465267    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 14:40:20.468368 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465269    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 14:40:20.468368 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465272    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 14:40:20.468368 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465274    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 14:40:20.468368 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465277    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 14:40:20.468368 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465280    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 14:40:20.468368 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465283    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 14:40:20.468368 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465287    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 14:40:20.468368 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465290    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 14:40:20.468368 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465293    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 14:40:20.468912 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465296    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 14:40:20.468912 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465299    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 14:40:20.468912 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465302    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 14:40:20.468912 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465305    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 14:40:20.468912 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465308    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 14:40:20.468912 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465311    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 14:40:20.468912 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465314    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 14:40:20.468912 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465318    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 14:40:20.468912 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465322    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 14:40:20.468912 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465325    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 14:40:20.468912 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465328    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 14:40:20.468912 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465331    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 14:40:20.468912 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465334    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 14:40:20.468912 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465337    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 14:40:20.468912 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465340    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 14:40:20.468912 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465343    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 14:40:20.468912 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465345    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 14:40:20.468912 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465348    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 14:40:20.468912 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465351    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 14:40:20.469376 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465353    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 14:40:20.469376 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465356    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 14:40:20.469376 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465358    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 14:40:20.469376 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465361    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 14:40:20.469376 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465363    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 14:40:20.469376 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465365    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 14:40:20.469376 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465368    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 14:40:20.469376 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465370    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 14:40:20.469376 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465373    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 14:40:20.469376 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465376    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 14:40:20.469376 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465378    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 14:40:20.469376 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465381    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 14:40:20.469376 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465383    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 14:40:20.469376 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465386    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 14:40:20.469376 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465388    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 14:40:20.469376 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465391    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 14:40:20.469376 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465394    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 14:40:20.469376 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465396    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 14:40:20.469376 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465399    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 14:40:20.469376 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465402    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 14:40:20.469899 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465404    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 14:40:20.469899 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465407    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 14:40:20.469899 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465409    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 14:40:20.469899 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465412    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 14:40:20.469899 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465415    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 14:40:20.469899 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465417    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 14:40:20.469899 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465419    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 14:40:20.469899 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465423    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 14:40:20.469899 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465425    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 14:40:20.469899 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465428    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 14:40:20.469899 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465431    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 14:40:20.469899 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465433    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 14:40:20.469899 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465436    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 14:40:20.469899 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.465441    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 14:40:20.469899 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465572    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 14:40:20.470269 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465577    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 14:40:20.470269 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465580    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 14:40:20.470269 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465583    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 14:40:20.470269 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465586    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 14:40:20.470269 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465589    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 14:40:20.470269 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465592    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 14:40:20.470269 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465595    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 14:40:20.470269 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465598    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 14:40:20.470269 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465601    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 14:40:20.470269 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465604    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 14:40:20.470269 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465607    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 14:40:20.470269 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465610    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 14:40:20.470269 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465613    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 14:40:20.470269 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465615    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 14:40:20.470269 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465618    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 14:40:20.470269 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465621    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 14:40:20.470269 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465624    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 14:40:20.470269 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465626    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 14:40:20.470269 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465629    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 14:40:20.470269 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465632    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 14:40:20.470741 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465635    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 14:40:20.470741 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465638    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 14:40:20.470741 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465641    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 14:40:20.470741 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465643    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 14:40:20.470741 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465646    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 14:40:20.470741 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465648    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 14:40:20.470741 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465651    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 14:40:20.470741 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465654    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 14:40:20.470741 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465657    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 14:40:20.470741 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465659    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 14:40:20.470741 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465662    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 14:40:20.470741 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465664    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 14:40:20.470741 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465667    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 14:40:20.470741 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465669    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 14:40:20.470741 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465672    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 14:40:20.470741 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465675    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 14:40:20.470741 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465677    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 14:40:20.470741 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465680    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 14:40:20.470741 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465682    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 14:40:20.470741 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465685    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 14:40:20.471336 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465687    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 14:40:20.471336 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465690    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 14:40:20.471336 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465692    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 14:40:20.471336 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465695    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 14:40:20.471336 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465697    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 14:40:20.471336 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465700    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 14:40:20.471336 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465702    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 14:40:20.471336 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465705    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 14:40:20.471336 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465707    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 14:40:20.471336 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465710    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 14:40:20.471336 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465714    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 14:40:20.471336 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465718    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 14:40:20.471336 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465721    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 14:40:20.471336 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465724    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 14:40:20.471336 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465727    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 14:40:20.471336 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465730    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 14:40:20.471336 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465732    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 14:40:20.471336 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465735    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 14:40:20.471336 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465738    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 14:40:20.471788 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465741    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 14:40:20.471788 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465745    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 14:40:20.471788 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465748    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 14:40:20.471788 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465750    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 14:40:20.471788 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465753    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 14:40:20.471788 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465756    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 14:40:20.471788 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465758    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 14:40:20.471788 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465761    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 14:40:20.471788 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465763    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 14:40:20.471788 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465766    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 14:40:20.471788 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465768    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 14:40:20.471788 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465771    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 14:40:20.471788 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465773    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 14:40:20.471788 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465776    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 14:40:20.471788 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465778    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 14:40:20.471788 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465781    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 14:40:20.471788 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465783    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 14:40:20.471788 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465786    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 14:40:20.471788 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465788    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 14:40:20.472254 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465791    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 14:40:20.472254 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465793    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 14:40:20.472254 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465796    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 14:40:20.472254 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465799    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 14:40:20.472254 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465801    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 14:40:20.472254 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465805    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 14:40:20.472254 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465809    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 14:40:20.472254 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465812    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 14:40:20.472254 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465831    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 14:40:20.472254 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465834    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 14:40:20.472254 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465837    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 14:40:20.472254 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465839    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 14:40:20.472254 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465842    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 14:40:20.472254 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465845    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 14:40:20.472254 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465847    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 14:40:20.472254 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465850    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 14:40:20.472254 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465853    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 14:40:20.472254 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465856    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 14:40:20.472254 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465858    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 14:40:20.472725 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:20.465861    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 14:40:20.472725 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.465865    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 14:40:20.472725 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.465967    2568 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 14:40:20.472725 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.469460    2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 14:40:20.472725 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.470321    2568 server.go:1019] "Starting client certificate rotation"
Sep 11 14:40:20.472725 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.470423    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 14:40:20.472725 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.470456    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 14:40:20.472725 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.472549    2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 14:40:20.473699 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.473678    2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 14:40:20.485354 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.485337    2568 log.go:25] "Validated CRI v1 runtime API"
Sep 11 14:40:20.488067 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.488052    2568 log.go:25] "Validated CRI v1 image API"
Sep 11 14:40:20.488885 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.488869    2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 14:40:20.491626 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.491604    2568 fs.go:135] Filesystem UUIDs: map[5e369445-d527-4d09-84f0-1793e2b71c58:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9785352d-7d1d-428c-9b5d-3bf92ac4962a:/dev/nvme0n1p3]
Sep 11 14:40:20.491682 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.491626    2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 14:40:20.497036 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.496932    2568 manager.go:216] Machine: {Timestamp:2026-09-11 14:40:20.495728244 +0000 UTC m=+0.283992534 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099404 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24379ab741e39e9c027d0138e08d54 SystemUUID:ec24379a-b741-e39e-9c02-7d0138e08d54 BootID:38176bff-0785-4b3b-a3de-232d4badecef Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8f:86:85:56:df Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8f:86:85:56:df Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:25:f5:2a:4c:33 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 14:40:20.497036 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.497034    2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 14:40:20.497142 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.497106    2568 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 14:40:20.499935 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.499907    2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 14:40:20.500068 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.499940    2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-126.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 14:40:20.500584 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.500575    2568 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 14:40:20.500613 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.500587    2568 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 14:40:20.500613 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.500595    2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 14:40:20.501209 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.501199    2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 14:40:20.502302 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.502291    2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 14:40:20.504966 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.504950    2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 14:40:20.506249 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.506238    2568 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 14:40:20.506348 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.506339    2568 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 14:40:20.506378 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.506367    2568 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 14:40:20.506378 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.506377    2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 14:40:20.506433 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.506390    2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 14:40:20.506433 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.506400    2568 kubelet.go:398] "Adding apiserver pod source"
Sep 11 14:40:20.506433 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.506409    2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 14:40:20.506812 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.506795    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 14:40:20.506896 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.506877    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 14:40:20.508752 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.508737    2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 14:40:20.511743 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.511725    2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 14:40:20.511834 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.511757    2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 14:40:20.514502 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.514486    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 14:40:20.514573 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.514506    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 14:40:20.514573 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.514513    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 14:40:20.514573 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.514519    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 14:40:20.514573 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.514527    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 14:40:20.514573 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.514533    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 14:40:20.514573 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.514570    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 14:40:20.514573 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.514576    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 14:40:20.514806 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.514583    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 14:40:20.514806 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.514589    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 14:40:20.514806 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.514598    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 14:40:20.514806 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.514603    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 14:40:20.514806 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.514626    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 14:40:20.514806 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.514631    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 14:40:20.517730 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.517682    2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-126.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 14:40:20.517844 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:20.517731    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 14:40:20.517904 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:20.517839    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-126.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 14:40:20.518207 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.518195    2568 server.go:1268] "Started kubelet"
Sep 11 14:40:20.518317 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.518280    2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 14:40:20.518317 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.518275    2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 14:40:20.518452 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.518344    2568 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 14:40:20.519129 ip-10-0-133-126 systemd[1]: Started Kubernetes Kubelet.
Sep 11 14:40:20.520181 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.519485    2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 14:40:20.520564 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.520548    2568 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 14:40:20.523010 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:20.522978    2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 14:40:20.524991 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.524966    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 14:40:20.525461 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.525441    2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 14:40:20.525626 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:20.525562    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-126.ec2.internal\" not found"
Sep 11 14:40:20.526111 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.526088    2568 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 14:40:20.526111 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.526100    2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 14:40:20.526111 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.526108    2568 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 14:40:20.526271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.526207    2568 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 14:40:20.526271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.526214    2568 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 14:40:20.526349 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.526272    2568 factory.go:55] Registering systemd factory
Sep 11 14:40:20.526349 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.526286    2568 factory.go:223] Registration of the systemd container factory successfully
Sep 11 14:40:20.526519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.526506    2568 factory.go:153] Registering CRI-O factory
Sep 11 14:40:20.526519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.526520    2568 factory.go:223] Registration of the crio container factory successfully
Sep 11 14:40:20.526583 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.526565    2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 14:40:20.526583 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.526581    2568 factory.go:103] Registering Raw factory
Sep 11 14:40:20.526655 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.526594    2568 manager.go:1196] Started watching for new ooms in manager
Sep 11 14:40:20.527003 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.526991    2568 manager.go:349] Starting recovery of all containers
Sep 11 14:40:20.529033 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:20.529004    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 14:40:20.529130 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:20.529078    2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-126.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 14:40:20.529671 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:20.528797    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-126.ec2.internal.18d44b2fc545efb2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-126.ec2.internal,UID:ip-10-0-133-126.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-126.ec2.internal,},FirstTimestamp:2026-09-11 14:40:20.518170546 +0000 UTC m=+0.306434836,LastTimestamp:2026-09-11 14:40:20.518170546 +0000 UTC m=+0.306434836,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-126.ec2.internal,}"
Sep 11 14:40:20.537154 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.536995    2568 manager.go:354] Recovery completed
Sep 11 14:40:20.539445 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.539420    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2trvl"
Sep 11 14:40:20.543025 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.543007    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:40:20.545013 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.544996    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:40:20.545075 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.545026    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:40:20.545075 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.545039    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:40:20.545570 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.545557    2568 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 14:40:20.545613 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.545570    2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 14:40:20.545613 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.545575    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:40:20.545613 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.545585    2568 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 14:40:20.546467 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.546446    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:40:20.546539 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.546480    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:40:20.546539 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.546493    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:40:20.547210 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:20.547143    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-126.ec2.internal.18d44b2fc6df8574  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-126.ec2.internal,UID:ip-10-0-133-126.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-126.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-126.ec2.internal,},FirstTimestamp:2026-09-11 14:40:20.545013108 +0000 UTC m=+0.333277401,LastTimestamp:2026-09-11 14:40:20.545013108 +0000 UTC m=+0.333277401,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-126.ec2.internal,}"
Sep 11 14:40:20.548085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.548071    2568 policy_none.go:49] "None policy: Start"
Sep 11 14:40:20.548153 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.548089    2568 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 14:40:20.548153 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.548102    2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 14:40:20.548746 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.548725    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2trvl"
Sep 11 14:40:20.550098 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.550084    2568 policy_none.go:47] "Start"
Sep 11 14:40:20.586394 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.586377    2568 manager.go:337] "Starting Device Plugin manager"
Sep 11 14:40:20.607158 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:20.586414    2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 14:40:20.607158 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.586425    2568 server.go:87] "Starting device plugin registration server"
Sep 11 14:40:20.607158 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.586751    2568 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 14:40:20.607158 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.586762    2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 14:40:20.607158 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.586853    2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 14:40:20.607158 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.586922    2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 14:40:20.607158 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.586930    2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 14:40:20.607158 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:20.587434    2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 14:40:20.607158 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:20.587486    2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-126.ec2.internal\" not found"
Sep 11 14:40:20.653652 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.653622    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 14:40:20.654755 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.654735    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 14:40:20.654809 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.654798    2568 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 14:40:20.654867 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.654843    2568 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 14:40:20.654961 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:20.654942    2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 14:40:20.657398 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.657379    2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 14:40:20.687503 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.687484    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:40:20.688419 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.688398    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:40:20.688517 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.688433    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:40:20.688517 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.688449    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:40:20.688517 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.688478    2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-126.ec2.internal"
Sep 11 14:40:20.697079 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.697015    2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-126.ec2.internal"
Sep 11 14:40:20.697079 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:20.697036    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-126.ec2.internal\": node \"ip-10-0-133-126.ec2.internal\" not found"
Sep 11 14:40:20.716453 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:20.716436    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-126.ec2.internal\" not found"
Sep 11 14:40:20.755487 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.755464    2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-126.ec2.internal"]
Sep 11 14:40:20.755568 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.755526    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:40:20.756383 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.756367    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:40:20.756458 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.756397    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:40:20.756458 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.756411    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:40:20.758722 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.758710    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:40:20.758887 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.758873    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal"
Sep 11 14:40:20.758938 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.758904    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:40:20.759446 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.759423    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:40:20.759446 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.759434    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:40:20.759559 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.759452    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:40:20.759559 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.759470    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:40:20.759559 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.759452    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:40:20.759559 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.759545    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:40:20.761763 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.761749    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-126.ec2.internal"
Sep 11 14:40:20.761825 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.761772    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:40:20.762375 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.762362    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:40:20.762436 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.762386    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:40:20.762436 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.762400    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:40:20.790381 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:20.790362    2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-126.ec2.internal\" not found" node="ip-10-0-133-126.ec2.internal"
Sep 11 14:40:20.794632 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:20.794618    2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-126.ec2.internal\" not found" node="ip-10-0-133-126.ec2.internal"
Sep 11 14:40:20.816495 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:20.816472    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-126.ec2.internal\" not found"
Sep 11 14:40:20.828164 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.828145    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/12cf59dc053d83a81bcce4c580b849cb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal\" (UID: \"12cf59dc053d83a81bcce4c580b849cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal"
Sep 11 14:40:20.828228 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.828168    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12cf59dc053d83a81bcce4c580b849cb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal\" (UID: \"12cf59dc053d83a81bcce4c580b849cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal"
Sep 11 14:40:20.828228 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.828184    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c006646ef20c08a2b2e3dc82507ab9d-config\") pod \"kube-apiserver-proxy-ip-10-0-133-126.ec2.internal\" (UID: \"1c006646ef20c08a2b2e3dc82507ab9d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-126.ec2.internal"
Sep 11 14:40:20.916934 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:20.916911    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-126.ec2.internal\" not found"
Sep 11 14:40:20.928286 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.928265    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12cf59dc053d83a81bcce4c580b849cb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal\" (UID: \"12cf59dc053d83a81bcce4c580b849cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal"
Sep 11 14:40:20.928362 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.928292    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c006646ef20c08a2b2e3dc82507ab9d-config\") pod \"kube-apiserver-proxy-ip-10-0-133-126.ec2.internal\" (UID: \"1c006646ef20c08a2b2e3dc82507ab9d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-126.ec2.internal"
Sep 11 14:40:20.928362 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.928309    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/12cf59dc053d83a81bcce4c580b849cb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal\" (UID: \"12cf59dc053d83a81bcce4c580b849cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal"
Sep 11 14:40:20.928441 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.928362    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12cf59dc053d83a81bcce4c580b849cb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal\" (UID: \"12cf59dc053d83a81bcce4c580b849cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal"
Sep 11 14:40:20.928441 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.928372    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/12cf59dc053d83a81bcce4c580b849cb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal\" (UID: \"12cf59dc053d83a81bcce4c580b849cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal"
Sep 11 14:40:20.928441 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:20.928395    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c006646ef20c08a2b2e3dc82507ab9d-config\") pod \"kube-apiserver-proxy-ip-10-0-133-126.ec2.internal\" (UID: \"1c006646ef20c08a2b2e3dc82507ab9d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-126.ec2.internal"
Sep 11 14:40:21.017784 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:21.017701    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-126.ec2.internal\" not found"
Sep 11 14:40:21.093295 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:21.093263    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal"
Sep 11 14:40:21.097643 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:21.097626    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-126.ec2.internal"
Sep 11 14:40:21.118468 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:21.118442    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-126.ec2.internal\" not found"
Sep 11 14:40:21.220173 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:21.220140    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-126.ec2.internal\" not found"
Sep 11 14:40:21.321653 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:21.321617    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-126.ec2.internal\" not found"
Sep 11 14:40:21.422375 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:21.422335    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-126.ec2.internal\" not found"
Sep 11 14:40:21.472633 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:21.472599    2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 14:40:21.473091 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:21.472751    2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 14:40:21.523082 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:21.523051    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-126.ec2.internal\" not found"
Sep 11 14:40:21.554163 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:21.554120    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 14:35:20 +0000 UTC" deadline="2028-02-07 04:28:40.475643052 +0000 UTC"
Sep 11 14:40:21.554163 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:21.554156    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12325h48m18.921491911s"
Sep 11 14:40:21.623857 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:21.623783    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-126.ec2.internal\" not found"
Sep 11 14:40:21.724331 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:21.724290    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-126.ec2.internal\" not found"
Sep 11 14:40:21.801314 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:21.801276    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c006646ef20c08a2b2e3dc82507ab9d.slice/crio-652fd2d6f2394e43afb149bbd648c75bf3e5b423fb93bce9b8462215ea004f47 WatchSource:0}: Error finding container 652fd2d6f2394e43afb149bbd648c75bf3e5b423fb93bce9b8462215ea004f47: Status 404 returned error can't find the container with id 652fd2d6f2394e43afb149bbd648c75bf3e5b423fb93bce9b8462215ea004f47
Sep 11 14:40:21.801952 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:21.801927    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12cf59dc053d83a81bcce4c580b849cb.slice/crio-58a9596997dbc72dad2018ed6ebe14e852df8c620e21c4e0a216b0150fc4fadd WatchSource:0}: Error finding container 58a9596997dbc72dad2018ed6ebe14e852df8c620e21c4e0a216b0150fc4fadd: Status 404 returned error can't find the container with id 58a9596997dbc72dad2018ed6ebe14e852df8c620e21c4e0a216b0150fc4fadd
Sep 11 14:40:21.807848 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:21.807813    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:40:21.825017 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:21.824984    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-126.ec2.internal\" not found"
Sep 11 14:40:21.890399 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:21.890323    2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 14:40:21.926972 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:21.926933    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-126.ec2.internal\" not found"
Sep 11 14:40:21.986379 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:21.986353    2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 14:40:22.007406 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.007386    2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 14:40:22.026521 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.026493    2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal"
Sep 11 14:40:22.040659 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.040641    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 14:40:22.043446 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.041838    2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-126.ec2.internal"
Sep 11 14:40:22.053038 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.053023    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 14:40:22.508728 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.508692    2568 apiserver.go:52] "Watching apiserver"
Sep 11 14:40:22.519954 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.519927    2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 14:40:22.521289 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.521260    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-zsqws","openshift-network-diagnostics/network-check-target-qvvkt","openshift-network-operator/iptables-alerter-pvpvd","openshift-ovn-kubernetes/ovnkube-node-dg2rh","kube-system/global-pull-secret-syncer-sxv9g","kube-system/konnectivity-agent-x9q88","kube-system/kube-apiserver-proxy-ip-10-0-133-126.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft","openshift-image-registry/node-ca-7mlmr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal","openshift-multus/multus-additional-cni-plugins-nflgb","openshift-multus/multus-hhmbs","openshift-cluster-node-tuning-operator/tuned-vh5fh","openshift-dns/node-resolver-xzmzt"]
Sep 11 14:40:22.524231 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.524210    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.525742 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.525717    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 14:40:22.526632 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.526610    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:22.526729 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:22.526693    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvvkt" podUID="f2636cfe-b8a3-4c64-94f5-7026d6603aa0"
Sep 11 14:40:22.528349 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.528147    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wmmh4\""
Sep 11 14:40:22.528349 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.528154    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 14:40:22.528349 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.528154    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 14:40:22.529171 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.528730    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pvpvd"
Sep 11 14:40:22.532023 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.532000    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.535046 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.534635    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 14:40:22.535046 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.534917    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 14:40:22.535163 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.535151    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4vbsc\""
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.535265    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x9q88"
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.535358    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.535721    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.535802    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536041    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536285    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-sys\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536312    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-lib-modules\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536335    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-host\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536350    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536362    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkm7z\" (UniqueName: \"kubernetes.io/projected/b452c8a8-b01e-4250-bc67-2708e22918bb-kube-api-access-fkm7z\") pod \"iptables-alerter-pvpvd\" (UID: \"b452c8a8-b01e-4250-bc67-2708e22918bb\") " pod="openshift-network-operator/iptables-alerter-pvpvd"
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536378    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ddcg5\""
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536391    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-etc-kubernetes\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536413    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-run\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:22.536414    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsqws" podUID="1cb940cd-c21b-4118-ad56-bb823c730604"
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536434    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-var-lib-kubelet\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536458    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxglz\" (UniqueName: \"kubernetes.io/projected/c974f271-da27-406b-9d10-4c1889517ce1-kube-api-access-hxglz\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536482    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txk6q\" (UniqueName: \"kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q\") pod \"network-check-target-qvvkt\" (UID: \"f2636cfe-b8a3-4c64-94f5-7026d6603aa0\") " pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536508    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b452c8a8-b01e-4250-bc67-2708e22918bb-iptables-alerter-script\") pod \"iptables-alerter-pvpvd\" (UID: \"b452c8a8-b01e-4250-bc67-2708e22918bb\") " pod="openshift-network-operator/iptables-alerter-pvpvd"
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536576    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-etc-sysctl-conf\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536583    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 14:40:22.537975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536670    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 14:40:22.539126 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536593    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-etc-systemd\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.539126 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536720    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c974f271-da27-406b-9d10-4c1889517ce1-etc-tuned\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.539126 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536746    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c974f271-da27-406b-9d10-4c1889517ce1-tmp\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.539126 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536770    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b452c8a8-b01e-4250-bc67-2708e22918bb-host-slash\") pod \"iptables-alerter-pvpvd\" (UID: \"b452c8a8-b01e-4250-bc67-2708e22918bb\") " pod="openshift-network-operator/iptables-alerter-pvpvd"
Sep 11 14:40:22.539126 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536801    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-etc-modprobe-d\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.539126 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536826    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-etc-sysconfig\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.539126 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536870    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-etc-sysctl-d\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.539126 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.536910    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 14:40:22.539126 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.538997    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.539637 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.539516    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 14:40:22.541663 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.539923    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dn9nk\""
Sep 11 14:40:22.541663 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.540025    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 14:40:22.541663 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.541552    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7mlmr"
Sep 11 14:40:22.543799 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.543773    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xzmzt"
Sep 11 14:40:22.546969 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.546420    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.547068 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.547016    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 14:40:22.547068 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.547028    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 14:40:22.547433 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.547199    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 14:40:22.547433 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.547347    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 14:40:22.548006 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.547664    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 14:40:22.548006 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.547776    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gdc62\""
Sep 11 14:40:22.548209 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.548082    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 14:40:22.550084 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.550058    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-29mf9\""
Sep 11 14:40:22.550290 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.550262    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q9nd6\""
Sep 11 14:40:22.552777 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.550759    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 14:40:22.553480 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.553456    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 14:40:22.553850 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.553830    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 14:40:22.554283 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.554260    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 14:40:22.554438 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.554419    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 14:40:22.554548 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.554525    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-464jn\""
Sep 11 14:40:22.555385 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.555015    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.555385 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.554696    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 14:40:22.555385 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.555277    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 14:40:22.556966 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.556661    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:22.556966 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:22.556745    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxv9g" podUID="3296271a-8ecd-483d-824e-3b9896f81e3c"
Sep 11 14:40:22.561821 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.559715    2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 14:40:22.561821 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.561013    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 14:40:22.561821 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.561121    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cwmpv\""
Sep 11 14:40:22.607103 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.607077    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m48rb"
Sep 11 14:40:22.622533 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.622510    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m48rb"
Sep 11 14:40:22.627137 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.627115    2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 14:40:22.637550 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.637523    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6vmz\" (UniqueName: \"kubernetes.io/projected/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-kube-api-access-z6vmz\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.637659 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.637562    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-host-run-multus-certs\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.637659 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.637590    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-etc-kubernetes\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.637659 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.637626    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64bef023-3ded-4c9d-9e40-1b202e960fe2-tmp-dir\") pod \"node-resolver-xzmzt\" (UID: \"64bef023-3ded-4c9d-9e40-1b202e960fe2\") " pod="openshift-dns/node-resolver-xzmzt"
Sep 11 14:40:22.637659 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.637653    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.637818 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.637695    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-etc-kubernetes\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.637818 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.637749    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txk6q\" (UniqueName: \"kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q\") pod \"network-check-target-qvvkt\" (UID: \"f2636cfe-b8a3-4c64-94f5-7026d6603aa0\") " pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:22.637818 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.637796    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-etc-kubernetes\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.637818 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.637809    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-run-ovn-kubernetes\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.638030 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.637901    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-cnibin\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.638030 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.637949    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-host-var-lib-cni-multus\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.638030 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.637979    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-registration-dir\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.638030 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638009    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-etc-sysctl-conf\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.638230 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638036    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-etc-systemd\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.638230 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638063    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c974f271-da27-406b-9d10-4c1889517ce1-etc-tuned\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.638230 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638087    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b452c8a8-b01e-4250-bc67-2708e22918bb-host-slash\") pod \"iptables-alerter-pvpvd\" (UID: \"b452c8a8-b01e-4250-bc67-2708e22918bb\") " pod="openshift-network-operator/iptables-alerter-pvpvd"
Sep 11 14:40:22.638230 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638112    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-ovn-node-metrics-cert\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.638230 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638139    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-host-var-lib-cni-bin\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.638230 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638146    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-etc-systemd\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.638230 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638163    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-hostroot\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.638230 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638199    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-etc-modprobe-d\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.638230 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638215    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-etc-sysctl-conf\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.638230 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638226    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-etc-sysconfig\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.638759 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638265    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-multus-daemon-config\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.638759 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638277    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-etc-sysconfig\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.638759 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638293    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zznrk\" (UniqueName: \"kubernetes.io/projected/2695284a-7a7f-42ef-9f9a-e152c61edaf3-kube-api-access-zznrk\") pod \"node-ca-7mlmr\" (UID: \"2695284a-7a7f-42ef-9f9a-e152c61edaf3\") " pod="openshift-image-registry/node-ca-7mlmr"
Sep 11 14:40:22.638759 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638321    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-run-netns\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.638759 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638345    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-log-socket\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.638759 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638362    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-etc-modprobe-d\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.638759 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638369    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-ovnkube-config\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.638759 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638394    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-etc-selinux\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.638759 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638420    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6f17c87-b5e8-4a52-a3b4-f5b409f3e75f-agent-certs\") pod \"konnectivity-agent-x9q88\" (UID: \"e6f17c87-b5e8-4a52-a3b4-f5b409f3e75f\") " pod="kube-system/konnectivity-agent-x9q88"
Sep 11 14:40:22.638759 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638450    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-var-lib-kubelet\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.638759 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638475    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-run-ovn\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.638759 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638500    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qznws\" (UniqueName: \"kubernetes.io/projected/1cb940cd-c21b-4118-ad56-bb823c730604-kube-api-access-qznws\") pod \"network-metrics-daemon-zsqws\" (UID: \"1cb940cd-c21b-4118-ad56-bb823c730604\") " pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:22.638759 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638503    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-var-lib-kubelet\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.638759 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638423    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b452c8a8-b01e-4250-bc67-2708e22918bb-host-slash\") pod \"iptables-alerter-pvpvd\" (UID: \"b452c8a8-b01e-4250-bc67-2708e22918bb\") " pod="openshift-network-operator/iptables-alerter-pvpvd"
Sep 11 14:40:22.638759 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638528    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.638759 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638552    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-sys-fs\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.638759 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638576    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-cni-binary-copy\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.639566 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638599    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xff8v\" (UniqueName: \"kubernetes.io/projected/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-kube-api-access-xff8v\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.639566 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638646    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-run-openvswitch\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.639566 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638666    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-host-run-k8s-cni-cncf-io\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.639566 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638684    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-systemd-units\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.639566 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638699    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-slash\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.639566 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638715    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-node-log\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.639566 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638731    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-sys\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.639566 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638745    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-lib-modules\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.639566 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638763    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkm7z\" (UniqueName: \"kubernetes.io/projected/b452c8a8-b01e-4250-bc67-2708e22918bb-kube-api-access-fkm7z\") pod \"iptables-alerter-pvpvd\" (UID: \"b452c8a8-b01e-4250-bc67-2708e22918bb\") " pod="openshift-network-operator/iptables-alerter-pvpvd"
Sep 11 14:40:22.639566 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638789    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp5gx\" (UniqueName: \"kubernetes.io/projected/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-kube-api-access-vp5gx\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.639566 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638820    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-multus-cni-dir\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.639566 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638844    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2695284a-7a7f-42ef-9f9a-e152c61edaf3-host\") pod \"node-ca-7mlmr\" (UID: \"2695284a-7a7f-42ef-9f9a-e152c61edaf3\") " pod="openshift-image-registry/node-ca-7mlmr"
Sep 11 14:40:22.639566 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638879    2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 14:40:22.639566 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638904    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-cnibin\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.639566 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638933    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-multus-socket-dir-parent\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.639566 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638960    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hh2q\" (UniqueName: \"kubernetes.io/projected/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-kube-api-access-8hh2q\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.639566 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.638996    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c974f271-da27-406b-9d10-4c1889517ce1-tmp\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.640499 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639019    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-run-systemd\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.640499 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639044    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-cni-bin\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.640499 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639068    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-host-var-lib-kubelet\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.640499 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639107    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-multus-conf-dir\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.640499 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639135    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret\") pod \"global-pull-secret-syncer-sxv9g\" (UID: \"3296271a-8ecd-483d-824e-3b9896f81e3c\") " pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:22.640499 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639164    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sffjt\" (UniqueName: \"kubernetes.io/projected/64bef023-3ded-4c9d-9e40-1b202e960fe2-kube-api-access-sffjt\") pod \"node-resolver-xzmzt\" (UID: \"64bef023-3ded-4c9d-9e40-1b202e960fe2\") " pod="openshift-dns/node-resolver-xzmzt"
Sep 11 14:40:22.640499 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639191    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-etc-openvswitch\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.640499 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639216    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-sys\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.640499 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639219    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-env-overrides\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.640499 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639263    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-system-cni-dir\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.640499 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639293    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.640499 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639346    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-etc-sysctl-d\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.640499 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639398    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-host\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.640499 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639425    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-socket-dir\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.640499 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639463    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-device-dir\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.640499 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639486    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-run\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.640499 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639510    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxglz\" (UniqueName: \"kubernetes.io/projected/c974f271-da27-406b-9d10-4c1889517ce1-kube-api-access-hxglz\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.641414 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639537    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b452c8a8-b01e-4250-bc67-2708e22918bb-iptables-alerter-script\") pod \"iptables-alerter-pvpvd\" (UID: \"b452c8a8-b01e-4250-bc67-2708e22918bb\") " pod="openshift-network-operator/iptables-alerter-pvpvd"
Sep 11 14:40:22.641414 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639591    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.641414 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639618    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6f17c87-b5e8-4a52-a3b4-f5b409f3e75f-konnectivity-ca\") pod \"konnectivity-agent-x9q88\" (UID: \"e6f17c87-b5e8-4a52-a3b4-f5b409f3e75f\") " pod="kube-system/konnectivity-agent-x9q88"
Sep 11 14:40:22.641414 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639632    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-etc-sysctl-d\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.641414 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639644    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2695284a-7a7f-42ef-9f9a-e152c61edaf3-serviceca\") pod \"node-ca-7mlmr\" (UID: \"2695284a-7a7f-42ef-9f9a-e152c61edaf3\") " pod="openshift-image-registry/node-ca-7mlmr"
Sep 11 14:40:22.641414 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639653    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-host\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.641414 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639669    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64bef023-3ded-4c9d-9e40-1b202e960fe2-hosts-file\") pod \"node-resolver-xzmzt\" (UID: \"64bef023-3ded-4c9d-9e40-1b202e960fe2\") " pod="openshift-dns/node-resolver-xzmzt"
Sep 11 14:40:22.641414 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639692    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-var-lib-openvswitch\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.641414 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639735    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-host-run-netns\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.641414 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639741    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-lib-modules\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.641414 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639742    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c974f271-da27-406b-9d10-4c1889517ce1-run\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.641414 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639786    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3296271a-8ecd-483d-824e-3b9896f81e3c-kubelet-config\") pod \"global-pull-secret-syncer-sxv9g\" (UID: \"3296271a-8ecd-483d-824e-3b9896f81e3c\") " pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:22.641414 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639829    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3296271a-8ecd-483d-824e-3b9896f81e3c-dbus\") pod \"global-pull-secret-syncer-sxv9g\" (UID: \"3296271a-8ecd-483d-824e-3b9896f81e3c\") " pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:22.641414 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639873    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-system-cni-dir\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.641414 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639925    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.641414 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639950    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-cni-netd\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.641414 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.639982    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-ovnkube-script-lib\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.642224 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.640010    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs\") pod \"network-metrics-daemon-zsqws\" (UID: \"1cb940cd-c21b-4118-ad56-bb823c730604\") " pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:22.642224 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.640050    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-cni-binary-copy\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.642224 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.640085    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-kubelet\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.642224 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.640224    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b452c8a8-b01e-4250-bc67-2708e22918bb-iptables-alerter-script\") pod \"iptables-alerter-pvpvd\" (UID: \"b452c8a8-b01e-4250-bc67-2708e22918bb\") " pod="openshift-network-operator/iptables-alerter-pvpvd"
Sep 11 14:40:22.642838 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.642816    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c974f271-da27-406b-9d10-4c1889517ce1-etc-tuned\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.643238 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.643213    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c974f271-da27-406b-9d10-4c1889517ce1-tmp\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.646034 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:22.646011    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:40:22.646140 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:22.646040    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:40:22.646140 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:22.646055    2568 projected.go:196] Error preparing data for projected volume kube-api-access-txk6q for pod openshift-network-diagnostics/network-check-target-qvvkt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:22.646140 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:22.646130    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q podName:f2636cfe-b8a3-4c64-94f5-7026d6603aa0 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:23.146108604 +0000 UTC m=+2.934372903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-txk6q" (UniqueName: "kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q") pod "network-check-target-qvvkt" (UID: "f2636cfe-b8a3-4c64-94f5-7026d6603aa0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:22.648388 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.648348    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkm7z\" (UniqueName: \"kubernetes.io/projected/b452c8a8-b01e-4250-bc67-2708e22918bb-kube-api-access-fkm7z\") pod \"iptables-alerter-pvpvd\" (UID: \"b452c8a8-b01e-4250-bc67-2708e22918bb\") " pod="openshift-network-operator/iptables-alerter-pvpvd"
Sep 11 14:40:22.649944 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.649922    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxglz\" (UniqueName: \"kubernetes.io/projected/c974f271-da27-406b-9d10-4c1889517ce1-kube-api-access-hxglz\") pod \"tuned-vh5fh\" (UID: \"c974f271-da27-406b-9d10-4c1889517ce1\") " pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.660472 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.660423    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal" event={"ID":"12cf59dc053d83a81bcce4c580b849cb","Type":"ContainerStarted","Data":"58a9596997dbc72dad2018ed6ebe14e852df8c620e21c4e0a216b0150fc4fadd"}
Sep 11 14:40:22.662411 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.662383    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-126.ec2.internal" event={"ID":"1c006646ef20c08a2b2e3dc82507ab9d","Type":"ContainerStarted","Data":"652fd2d6f2394e43afb149bbd648c75bf3e5b423fb93bce9b8462215ea004f47"}
Sep 11 14:40:22.740328 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740298    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-etc-openvswitch\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.740493 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740337    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-env-overrides\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.740493 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740398    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-etc-openvswitch\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.740493 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740439    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-system-cni-dir\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.740493 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740465    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.740493 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740484    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-socket-dir\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.740775 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740501    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-device-dir\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.740775 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740519    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.740775 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740582    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.740775 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740598    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-device-dir\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.740775 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740616    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6f17c87-b5e8-4a52-a3b4-f5b409f3e75f-konnectivity-ca\") pod \"konnectivity-agent-x9q88\" (UID: \"e6f17c87-b5e8-4a52-a3b4-f5b409f3e75f\") " pod="kube-system/konnectivity-agent-x9q88"
Sep 11 14:40:22.740775 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740641    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2695284a-7a7f-42ef-9f9a-e152c61edaf3-serviceca\") pod \"node-ca-7mlmr\" (UID: \"2695284a-7a7f-42ef-9f9a-e152c61edaf3\") " pod="openshift-image-registry/node-ca-7mlmr"
Sep 11 14:40:22.740775 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740660    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-socket-dir\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.740775 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740668    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64bef023-3ded-4c9d-9e40-1b202e960fe2-hosts-file\") pod \"node-resolver-xzmzt\" (UID: \"64bef023-3ded-4c9d-9e40-1b202e960fe2\") " pod="openshift-dns/node-resolver-xzmzt"
Sep 11 14:40:22.740775 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740711    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-var-lib-openvswitch\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.740775 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740723    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64bef023-3ded-4c9d-9e40-1b202e960fe2-hosts-file\") pod \"node-resolver-xzmzt\" (UID: \"64bef023-3ded-4c9d-9e40-1b202e960fe2\") " pod="openshift-dns/node-resolver-xzmzt"
Sep 11 14:40:22.740775 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740640    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-system-cni-dir\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.740775 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740737    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-host-run-netns\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.740775 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740773    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-var-lib-openvswitch\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.740775 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740781    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3296271a-8ecd-483d-824e-3b9896f81e3c-kubelet-config\") pod \"global-pull-secret-syncer-sxv9g\" (UID: \"3296271a-8ecd-483d-824e-3b9896f81e3c\") " pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:22.741494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740807    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3296271a-8ecd-483d-824e-3b9896f81e3c-dbus\") pod \"global-pull-secret-syncer-sxv9g\" (UID: \"3296271a-8ecd-483d-824e-3b9896f81e3c\") " pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:22.741494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740814    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-host-run-netns\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.741494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740833    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-system-cni-dir\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.741494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740884    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.741494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740901    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3296271a-8ecd-483d-824e-3b9896f81e3c-dbus\") pod \"global-pull-secret-syncer-sxv9g\" (UID: \"3296271a-8ecd-483d-824e-3b9896f81e3c\") " pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:22.741494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740891    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3296271a-8ecd-483d-824e-3b9896f81e3c-kubelet-config\") pod \"global-pull-secret-syncer-sxv9g\" (UID: \"3296271a-8ecd-483d-824e-3b9896f81e3c\") " pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:22.741494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740909    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-cni-netd\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.741494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740951    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-cni-netd\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.741494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740961    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-ovnkube-script-lib\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.741494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.740989    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs\") pod \"network-metrics-daemon-zsqws\" (UID: \"1cb940cd-c21b-4118-ad56-bb823c730604\") " pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:22.741494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741005    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-system-cni-dir\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.741494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741005    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-env-overrides\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.741494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741016    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-cni-binary-copy\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.741494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741043    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-kubelet\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.741494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741066    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6vmz\" (UniqueName: \"kubernetes.io/projected/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-kube-api-access-z6vmz\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.741494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741089    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-host-run-multus-certs\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.741494 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:22.741116    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:40:22.741494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741123    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-etc-kubernetes\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.742463 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741146    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64bef023-3ded-4c9d-9e40-1b202e960fe2-tmp-dir\") pod \"node-resolver-xzmzt\" (UID: \"64bef023-3ded-4c9d-9e40-1b202e960fe2\") " pod="openshift-dns/node-resolver-xzmzt"
Sep 11 14:40:22.742463 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:22.741177    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs podName:1cb940cd-c21b-4118-ad56-bb823c730604 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:23.241159163 +0000 UTC m=+3.029423464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs") pod "network-metrics-daemon-zsqws" (UID: "1cb940cd-c21b-4118-ad56-bb823c730604") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:40:22.742463 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741178    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.742463 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741202    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.742463 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741203    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2695284a-7a7f-42ef-9f9a-e152c61edaf3-serviceca\") pod \"node-ca-7mlmr\" (UID: \"2695284a-7a7f-42ef-9f9a-e152c61edaf3\") " pod="openshift-image-registry/node-ca-7mlmr"
Sep 11 14:40:22.742463 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741223    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6f17c87-b5e8-4a52-a3b4-f5b409f3e75f-konnectivity-ca\") pod \"konnectivity-agent-x9q88\" (UID: \"e6f17c87-b5e8-4a52-a3b4-f5b409f3e75f\") " pod="kube-system/konnectivity-agent-x9q88"
Sep 11 14:40:22.742463 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741249    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-run-ovn-kubernetes\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.742463 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741275    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-cnibin\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.742463 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741282    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-kubelet\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.742463 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741322    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-etc-kubernetes\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.742463 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741328    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-host-var-lib-cni-multus\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.742463 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741358    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-registration-dir\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.742463 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741391    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-ovn-node-metrics-cert\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.742463 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741416    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-host-var-lib-cni-bin\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.742463 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741443    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-hostroot\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.742463 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741452    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-host-run-multus-certs\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.742463 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741495    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-hostroot\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.743271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741506    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-cnibin\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.743271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741543    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-multus-daemon-config\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.743271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741572    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zznrk\" (UniqueName: \"kubernetes.io/projected/2695284a-7a7f-42ef-9f9a-e152c61edaf3-kube-api-access-zznrk\") pod \"node-ca-7mlmr\" (UID: \"2695284a-7a7f-42ef-9f9a-e152c61edaf3\") " pod="openshift-image-registry/node-ca-7mlmr"
Sep 11 14:40:22.743271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741598    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-run-netns\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.743271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741615    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64bef023-3ded-4c9d-9e40-1b202e960fe2-tmp-dir\") pod \"node-resolver-xzmzt\" (UID: \"64bef023-3ded-4c9d-9e40-1b202e960fe2\") " pod="openshift-dns/node-resolver-xzmzt"
Sep 11 14:40:22.743271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741624    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-log-socket\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.743271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741648    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-ovnkube-config\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.743271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741674    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-run-ovn-kubernetes\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.743271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741154    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.743271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741677    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-etc-selinux\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.743271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741716    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6f17c87-b5e8-4a52-a3b4-f5b409f3e75f-agent-certs\") pod \"konnectivity-agent-x9q88\" (UID: \"e6f17c87-b5e8-4a52-a3b4-f5b409f3e75f\") " pod="kube-system/konnectivity-agent-x9q88"
Sep 11 14:40:22.743271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741724    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-cni-binary-copy\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.743271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741742    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-run-ovn\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.743271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741746    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-host-var-lib-cni-bin\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.743271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741769    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qznws\" (UniqueName: \"kubernetes.io/projected/1cb940cd-c21b-4118-ad56-bb823c730604-kube-api-access-qznws\") pod \"network-metrics-daemon-zsqws\" (UID: \"1cb940cd-c21b-4118-ad56-bb823c730604\") " pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:22.743271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741780    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-etc-selinux\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.743271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741796    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.744085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741804    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-log-socket\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.744085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741822    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-sys-fs\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.744085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741846    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-cni-binary-copy\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.744085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741750    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-ovnkube-script-lib\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.744085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741847    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-run-netns\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.744085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741885    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xff8v\" (UniqueName: \"kubernetes.io/projected/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-kube-api-access-xff8v\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.744085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741932    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.744085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742003    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-sys-fs\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.744085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741934    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-run-openvswitch\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.744085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742064    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-host-run-k8s-cni-cncf-io\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.744085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742075    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.744085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742091    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-systemd-units\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.744085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742121    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-slash\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.744085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742129    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-host-var-lib-cni-multus\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.744085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742146    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-node-log\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.744085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742173    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-run-ovn\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.744085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742191    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp5gx\" (UniqueName: \"kubernetes.io/projected/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-kube-api-access-vp5gx\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.744875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742213    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-run-openvswitch\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.744875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742218    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-multus-cni-dir\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.744875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.741544    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-registration-dir\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.744875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742244    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2695284a-7a7f-42ef-9f9a-e152c61edaf3-host\") pod \"node-ca-7mlmr\" (UID: \"2695284a-7a7f-42ef-9f9a-e152c61edaf3\") " pod="openshift-image-registry/node-ca-7mlmr"
Sep 11 14:40:22.744875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742272    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-cnibin\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.744875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742282    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-slash\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.744875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742298    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-multus-socket-dir-parent\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.744875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742311    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-ovnkube-config\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.744875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742324    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-host-run-k8s-cni-cncf-io\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.744875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742324    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hh2q\" (UniqueName: \"kubernetes.io/projected/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-kube-api-access-8hh2q\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.744875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742358    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-systemd-units\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.744875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742367    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-run-systemd\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.744875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742365    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-multus-daemon-config\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.744875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742382    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-node-log\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.744875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742391    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-multus-cni-dir\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.744875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742418    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-cnibin\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.744875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742428    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-cni-binary-copy\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.744875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742446    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-multus-socket-dir-parent\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.745702 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742453    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-cni-bin\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.745702 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742474    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-run-systemd\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.745702 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742481    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-host-var-lib-kubelet\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.745702 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742507    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-multus-conf-dir\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.745702 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742515    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-host-cni-bin\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.745702 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742539    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret\") pod \"global-pull-secret-syncer-sxv9g\" (UID: \"3296271a-8ecd-483d-824e-3b9896f81e3c\") " pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:22.745702 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742568    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-host-var-lib-kubelet\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.745702 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742578    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-multus-conf-dir\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.745702 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742590    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2695284a-7a7f-42ef-9f9a-e152c61edaf3-host\") pod \"node-ca-7mlmr\" (UID: \"2695284a-7a7f-42ef-9f9a-e152c61edaf3\") " pod="openshift-image-registry/node-ca-7mlmr"
Sep 11 14:40:22.745702 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.742611    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sffjt\" (UniqueName: \"kubernetes.io/projected/64bef023-3ded-4c9d-9e40-1b202e960fe2-kube-api-access-sffjt\") pod \"node-resolver-xzmzt\" (UID: \"64bef023-3ded-4c9d-9e40-1b202e960fe2\") " pod="openshift-dns/node-resolver-xzmzt"
Sep 11 14:40:22.745702 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:22.744073    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:40:22.745702 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:22.744154    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret podName:3296271a-8ecd-483d-824e-3b9896f81e3c nodeName:}" failed. No retries permitted until 2026-09-11 14:40:23.244121082 +0000 UTC m=+3.032385381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret") pod "global-pull-secret-syncer-sxv9g" (UID: "3296271a-8ecd-483d-824e-3b9896f81e3c") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:40:22.745702 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.744867    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-ovn-node-metrics-cert\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.745702 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.744933    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6f17c87-b5e8-4a52-a3b4-f5b409f3e75f-agent-certs\") pod \"konnectivity-agent-x9q88\" (UID: \"e6f17c87-b5e8-4a52-a3b4-f5b409f3e75f\") " pod="kube-system/konnectivity-agent-x9q88"
Sep 11 14:40:22.751684 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.751654    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6vmz\" (UniqueName: \"kubernetes.io/projected/b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de-kube-api-access-z6vmz\") pod \"ovnkube-node-dg2rh\" (UID: \"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.753009 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.752982    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xff8v\" (UniqueName: \"kubernetes.io/projected/e57ff2ae-a4ff-4f18-a25a-f0ee3627e106-kube-api-access-xff8v\") pod \"multus-hhmbs\" (UID: \"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106\") " pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.753100 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.753071    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznws\" (UniqueName: \"kubernetes.io/projected/1cb940cd-c21b-4118-ad56-bb823c730604-kube-api-access-qznws\") pod \"network-metrics-daemon-zsqws\" (UID: \"1cb940cd-c21b-4118-ad56-bb823c730604\") " pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:22.754020 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.753998    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zznrk\" (UniqueName: \"kubernetes.io/projected/2695284a-7a7f-42ef-9f9a-e152c61edaf3-kube-api-access-zznrk\") pod \"node-ca-7mlmr\" (UID: \"2695284a-7a7f-42ef-9f9a-e152c61edaf3\") " pod="openshift-image-registry/node-ca-7mlmr"
Sep 11 14:40:22.754739 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.754714    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp5gx\" (UniqueName: \"kubernetes.io/projected/4caa60a7-ed08-4ca6-a4dd-df3e16edb938-kube-api-access-vp5gx\") pod \"aws-ebs-csi-driver-node-qg9ft\" (UID: \"4caa60a7-ed08-4ca6-a4dd-df3e16edb938\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.754811 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.754785    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sffjt\" (UniqueName: \"kubernetes.io/projected/64bef023-3ded-4c9d-9e40-1b202e960fe2-kube-api-access-sffjt\") pod \"node-resolver-xzmzt\" (UID: \"64bef023-3ded-4c9d-9e40-1b202e960fe2\") " pod="openshift-dns/node-resolver-xzmzt"
Sep 11 14:40:22.756070 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.756047    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hh2q\" (UniqueName: \"kubernetes.io/projected/53f3bf9a-6da4-46bb-a51e-f0c10c0da500-kube-api-access-8hh2q\") pod \"multus-additional-cni-plugins-nflgb\" (UID: \"53f3bf9a-6da4-46bb-a51e-f0c10c0da500\") " pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.843868 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.843820    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vh5fh"
Sep 11 14:40:22.860078 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.860055    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pvpvd"
Sep 11 14:40:22.873000 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.872978    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:22.880097 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.880074    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x9q88"
Sep 11 14:40:22.891027 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.891005    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft"
Sep 11 14:40:22.898131 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.898112    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7mlmr"
Sep 11 14:40:22.905906 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.905883    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xzmzt"
Sep 11 14:40:22.914733 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.914705    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nflgb"
Sep 11 14:40:22.921595 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.921577    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hhmbs"
Sep 11 14:40:22.986679 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:22.986654    2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 14:40:23.146290 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:23.146206    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txk6q\" (UniqueName: \"kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q\") pod \"network-check-target-qvvkt\" (UID: \"f2636cfe-b8a3-4c64-94f5-7026d6603aa0\") " pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:23.146440 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:23.146363    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:40:23.146440 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:23.146385    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:40:23.146440 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:23.146395    2568 projected.go:196] Error preparing data for projected volume kube-api-access-txk6q for pod openshift-network-diagnostics/network-check-target-qvvkt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:23.146552 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:23.146458    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q podName:f2636cfe-b8a3-4c64-94f5-7026d6603aa0 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:24.146439994 +0000 UTC m=+3.934704288 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-txk6q" (UniqueName: "kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q") pod "network-check-target-qvvkt" (UID: "f2636cfe-b8a3-4c64-94f5-7026d6603aa0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:23.247584 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:23.247544    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret\") pod \"global-pull-secret-syncer-sxv9g\" (UID: \"3296271a-8ecd-483d-824e-3b9896f81e3c\") " pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:23.247780 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:23.247607    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs\") pod \"network-metrics-daemon-zsqws\" (UID: \"1cb940cd-c21b-4118-ad56-bb823c730604\") " pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:23.247780 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:23.247665    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:40:23.247780 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:23.247715    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:40:23.247780 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:23.247731    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret podName:3296271a-8ecd-483d-824e-3b9896f81e3c nodeName:}" failed. No retries permitted until 2026-09-11 14:40:24.247712413 +0000 UTC m=+4.035976706 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret") pod "global-pull-secret-syncer-sxv9g" (UID: "3296271a-8ecd-483d-824e-3b9896f81e3c") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:40:23.247780 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:23.247769    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs podName:1cb940cd-c21b-4118-ad56-bb823c730604 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:24.247751882 +0000 UTC m=+4.036016175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs") pod "network-metrics-daemon-zsqws" (UID: "1cb940cd-c21b-4118-ad56-bb823c730604") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:40:23.627278 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:23.627228    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 14:35:22 +0000 UTC" deadline="2028-04-28 15:48:24.600951466 +0000 UTC"
Sep 11 14:40:23.627278 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:23.627274    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14281h8m0.973682197s"
Sep 11 14:40:23.838416 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:23.838228    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc974f271_da27_406b_9d10_4c1889517ce1.slice/crio-2982c51027ccafa853947c538f825684a05e78efc736712f38a5c75866edd879 WatchSource:0}: Error finding container 2982c51027ccafa853947c538f825684a05e78efc736712f38a5c75866edd879: Status 404 returned error can't find the container with id 2982c51027ccafa853947c538f825684a05e78efc736712f38a5c75866edd879
Sep 11 14:40:23.841403 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:23.841245    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2695284a_7a7f_42ef_9f9a_e152c61edaf3.slice/crio-06ddfbc36eed8ed2cc679c62c32030947fd269b0c1a8ef3e731213e0509add99 WatchSource:0}: Error finding container 06ddfbc36eed8ed2cc679c62c32030947fd269b0c1a8ef3e731213e0509add99: Status 404 returned error can't find the container with id 06ddfbc36eed8ed2cc679c62c32030947fd269b0c1a8ef3e731213e0509add99
Sep 11 14:40:23.843790 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:23.843769    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b0ac85_9e4b_4ec7_afdf_2a9082f8c6de.slice/crio-1c9649bac6708c28055458db40d63f2ce4e085df83881019ccb0284880f9e404 WatchSource:0}: Error finding container 1c9649bac6708c28055458db40d63f2ce4e085df83881019ccb0284880f9e404: Status 404 returned error can't find the container with id 1c9649bac6708c28055458db40d63f2ce4e085df83881019ccb0284880f9e404
Sep 11 14:40:23.850083 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:23.850061    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f17c87_b5e8_4a52_a3b4_f5b409f3e75f.slice/crio-8de75cdc75f4422baff8d7b08084d16e87dd2eb3fb8b115d3e1a6982efb34ad8 WatchSource:0}: Error finding container 8de75cdc75f4422baff8d7b08084d16e87dd2eb3fb8b115d3e1a6982efb34ad8: Status 404 returned error can't find the container with id 8de75cdc75f4422baff8d7b08084d16e87dd2eb3fb8b115d3e1a6982efb34ad8
Sep 11 14:40:24.153482 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.153422    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txk6q\" (UniqueName: \"kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q\") pod \"network-check-target-qvvkt\" (UID: \"f2636cfe-b8a3-4c64-94f5-7026d6603aa0\") " pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:24.153592 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:24.153574    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:40:24.153636 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:24.153597    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:40:24.153636 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:24.153608    2568 projected.go:196] Error preparing data for projected volume kube-api-access-txk6q for pod openshift-network-diagnostics/network-check-target-qvvkt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:24.153715 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:24.153656    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q podName:f2636cfe-b8a3-4c64-94f5-7026d6603aa0 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:26.153641753 +0000 UTC m=+5.941906035 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-txk6q" (UniqueName: "kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q") pod "network-check-target-qvvkt" (UID: "f2636cfe-b8a3-4c64-94f5-7026d6603aa0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:24.254422 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.254395    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret\") pod \"global-pull-secret-syncer-sxv9g\" (UID: \"3296271a-8ecd-483d-824e-3b9896f81e3c\") " pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:24.254528 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.254436    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs\") pod \"network-metrics-daemon-zsqws\" (UID: \"1cb940cd-c21b-4118-ad56-bb823c730604\") " pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:24.254528 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:24.254524    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:40:24.254608 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:24.254534    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:40:24.254608 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:24.254569    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs podName:1cb940cd-c21b-4118-ad56-bb823c730604 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:26.254556382 +0000 UTC m=+6.042820659 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs") pod "network-metrics-daemon-zsqws" (UID: "1cb940cd-c21b-4118-ad56-bb823c730604") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:40:24.254608 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:24.254585    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret podName:3296271a-8ecd-483d-824e-3b9896f81e3c nodeName:}" failed. No retries permitted until 2026-09-11 14:40:26.254573839 +0000 UTC m=+6.042838120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret") pod "global-pull-secret-syncer-sxv9g" (UID: "3296271a-8ecd-483d-824e-3b9896f81e3c") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:40:24.627782 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.627720    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 14:35:22 +0000 UTC" deadline="2028-05-08 15:48:34.068734794 +0000 UTC"
Sep 11 14:40:24.627782 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.627779    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14521h8m9.440959253s"
Sep 11 14:40:24.656970 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.656936    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:24.657114 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:24.657081    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvvkt" podUID="f2636cfe-b8a3-4c64-94f5-7026d6603aa0"
Sep 11 14:40:24.657488 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.657469    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:24.657586 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:24.657567    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxv9g" podUID="3296271a-8ecd-483d-824e-3b9896f81e3c"
Sep 11 14:40:24.657720 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.657634    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:24.657936 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:24.657726    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsqws" podUID="1cb940cd-c21b-4118-ad56-bb823c730604"
Sep 11 14:40:24.674470 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.674094    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-126.ec2.internal" event={"ID":"1c006646ef20c08a2b2e3dc82507ab9d","Type":"ContainerStarted","Data":"acaea37a65bd5efd6509350ff3287ba95a180173030e9177fecc4b8138e11a6c"}
Sep 11 14:40:24.686029 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.685999    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hhmbs" event={"ID":"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106","Type":"ContainerStarted","Data":"18707f62241bd04cb54bea32c2250c289ccaf44c6f08038e11e8f8596aba0716"}
Sep 11 14:40:24.697939 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.697908    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xzmzt" event={"ID":"64bef023-3ded-4c9d-9e40-1b202e960fe2","Type":"ContainerStarted","Data":"e3017aabc1bc73bca403d187da6d97ac2af75d5c8f67b6e9097ff786e32da91c"}
Sep 11 14:40:24.704653 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.704589    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x9q88" event={"ID":"e6f17c87-b5e8-4a52-a3b4-f5b409f3e75f","Type":"ContainerStarted","Data":"8de75cdc75f4422baff8d7b08084d16e87dd2eb3fb8b115d3e1a6982efb34ad8"}
Sep 11 14:40:24.708885 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.708787    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7mlmr" event={"ID":"2695284a-7a7f-42ef-9f9a-e152c61edaf3","Type":"ContainerStarted","Data":"06ddfbc36eed8ed2cc679c62c32030947fd269b0c1a8ef3e731213e0509add99"}
Sep 11 14:40:24.717569 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.717535    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vh5fh" event={"ID":"c974f271-da27-406b-9d10-4c1889517ce1","Type":"ContainerStarted","Data":"2982c51027ccafa853947c538f825684a05e78efc736712f38a5c75866edd879"}
Sep 11 14:40:24.732521 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.732399    2568 generic.go:356] "Generic (PLEG): container finished" podID="12cf59dc053d83a81bcce4c580b849cb" containerID="cdd78e579b961acdbd52567a3a2b3500697e9ee6c5b13c5fed3db2bf3a226531" exitCode=0
Sep 11 14:40:24.732521 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.732478    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal" event={"ID":"12cf59dc053d83a81bcce4c580b849cb","Type":"ContainerDied","Data":"cdd78e579b961acdbd52567a3a2b3500697e9ee6c5b13c5fed3db2bf3a226531"}
Sep 11 14:40:24.745911 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.745840    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nflgb" event={"ID":"53f3bf9a-6da4-46bb-a51e-f0c10c0da500","Type":"ContainerStarted","Data":"cd04036ac4eb363927d316fed2cbad81ca9f497096f3481c6316f5f05eb5fd11"}
Sep 11 14:40:24.750165 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.749271    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-126.ec2.internal" podStartSLOduration=2.74925401 podStartE2EDuration="2.74925401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:24.690287982 +0000 UTC m=+4.478552284" watchObservedRunningTime="2026-09-11 14:40:24.74925401 +0000 UTC m=+4.537518312"
Sep 11 14:40:24.752934 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.752907    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pvpvd" event={"ID":"b452c8a8-b01e-4250-bc67-2708e22918bb","Type":"ContainerStarted","Data":"33696248007bcd52a50bf4309a3e3446850788f7ac203248502c9149260d06b8"}
Sep 11 14:40:24.767595 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.767544    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft" event={"ID":"4caa60a7-ed08-4ca6-a4dd-df3e16edb938","Type":"ContainerStarted","Data":"4ef7acd5487cbe4c873ea1a059a5509726c447154ffcc6b0a4eb1d018127d14b"}
Sep 11 14:40:24.782253 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:24.782089    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh" event={"ID":"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de","Type":"ContainerStarted","Data":"1c9649bac6708c28055458db40d63f2ce4e085df83881019ccb0284880f9e404"}
Sep 11 14:40:25.801404 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:25.801319    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal" event={"ID":"12cf59dc053d83a81bcce4c580b849cb","Type":"ContainerStarted","Data":"e234580a8e257367c23646e38c3568567f43a18a6e3b82e147d7fd88e60b70bf"}
Sep 11 14:40:26.170183 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:26.170104    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txk6q\" (UniqueName: \"kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q\") pod \"network-check-target-qvvkt\" (UID: \"f2636cfe-b8a3-4c64-94f5-7026d6603aa0\") " pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:26.170337 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:26.170295    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:40:26.170337 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:26.170319    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:40:26.170337 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:26.170333    2568 projected.go:196] Error preparing data for projected volume kube-api-access-txk6q for pod openshift-network-diagnostics/network-check-target-qvvkt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:26.170505 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:26.170387    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q podName:f2636cfe-b8a3-4c64-94f5-7026d6603aa0 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:30.170368982 +0000 UTC m=+9.958633273 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-txk6q" (UniqueName: "kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q") pod "network-check-target-qvvkt" (UID: "f2636cfe-b8a3-4c64-94f5-7026d6603aa0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:26.271251 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:26.271214    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret\") pod \"global-pull-secret-syncer-sxv9g\" (UID: \"3296271a-8ecd-483d-824e-3b9896f81e3c\") " pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:26.271404 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:26.271280    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs\") pod \"network-metrics-daemon-zsqws\" (UID: \"1cb940cd-c21b-4118-ad56-bb823c730604\") " pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:26.271404 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:26.271323    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:40:26.271404 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:26.271390    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret podName:3296271a-8ecd-483d-824e-3b9896f81e3c nodeName:}" failed. No retries permitted until 2026-09-11 14:40:30.271372379 +0000 UTC m=+10.059636670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret") pod "global-pull-secret-syncer-sxv9g" (UID: "3296271a-8ecd-483d-824e-3b9896f81e3c") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:40:26.271404 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:26.271401    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:40:26.271620 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:26.271450    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs podName:1cb940cd-c21b-4118-ad56-bb823c730604 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:30.271435199 +0000 UTC m=+10.059699493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs") pod "network-metrics-daemon-zsqws" (UID: "1cb940cd-c21b-4118-ad56-bb823c730604") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:40:26.655939 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:26.655908    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:26.656121 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:26.656050    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxv9g" podUID="3296271a-8ecd-483d-824e-3b9896f81e3c"
Sep 11 14:40:26.656448 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:26.656425    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:26.656544 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:26.656527    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvvkt" podUID="f2636cfe-b8a3-4c64-94f5-7026d6603aa0"
Sep 11 14:40:26.656689 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:26.656669    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:26.656806 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:26.656786    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsqws" podUID="1cb940cd-c21b-4118-ad56-bb823c730604"
Sep 11 14:40:28.656957 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:28.656924    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:28.657421 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:28.657039    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvvkt" podUID="f2636cfe-b8a3-4c64-94f5-7026d6603aa0"
Sep 11 14:40:28.657421 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:28.657226    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:28.657421 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:28.657336    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsqws" podUID="1cb940cd-c21b-4118-ad56-bb823c730604"
Sep 11 14:40:28.657421 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:28.657404    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:28.657660 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:28.657482    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxv9g" podUID="3296271a-8ecd-483d-824e-3b9896f81e3c"
Sep 11 14:40:30.205080 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:30.205043    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txk6q\" (UniqueName: \"kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q\") pod \"network-check-target-qvvkt\" (UID: \"f2636cfe-b8a3-4c64-94f5-7026d6603aa0\") " pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:30.205529 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:30.205214    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:40:30.205529 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:30.205241    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:40:30.205529 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:30.205254    2568 projected.go:196] Error preparing data for projected volume kube-api-access-txk6q for pod openshift-network-diagnostics/network-check-target-qvvkt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:30.205529 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:30.205318    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q podName:f2636cfe-b8a3-4c64-94f5-7026d6603aa0 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:38.205300112 +0000 UTC m=+17.993564391 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-txk6q" (UniqueName: "kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q") pod "network-check-target-qvvkt" (UID: "f2636cfe-b8a3-4c64-94f5-7026d6603aa0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:30.305877 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:30.305793    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret\") pod \"global-pull-secret-syncer-sxv9g\" (UID: \"3296271a-8ecd-483d-824e-3b9896f81e3c\") " pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:30.306064 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:30.305960    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs\") pod \"network-metrics-daemon-zsqws\" (UID: \"1cb940cd-c21b-4118-ad56-bb823c730604\") " pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:30.306064 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:30.305965    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:40:30.306176 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:30.306074    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret podName:3296271a-8ecd-483d-824e-3b9896f81e3c nodeName:}" failed. No retries permitted until 2026-09-11 14:40:38.30605947 +0000 UTC m=+18.094323764 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret") pod "global-pull-secret-syncer-sxv9g" (UID: "3296271a-8ecd-483d-824e-3b9896f81e3c") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:40:30.306176 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:30.306086    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:40:30.306176 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:30.306144    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs podName:1cb940cd-c21b-4118-ad56-bb823c730604 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:38.306126786 +0000 UTC m=+18.094391075 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs") pod "network-metrics-daemon-zsqws" (UID: "1cb940cd-c21b-4118-ad56-bb823c730604") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:40:30.656740 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:30.656710    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:30.656918 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:30.656838    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvvkt" podUID="f2636cfe-b8a3-4c64-94f5-7026d6603aa0"
Sep 11 14:40:30.657049 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:30.657025    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:30.657182 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:30.657158    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsqws" podUID="1cb940cd-c21b-4118-ad56-bb823c730604"
Sep 11 14:40:30.657182 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:30.657176    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:30.657298 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:30.657269    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxv9g" podUID="3296271a-8ecd-483d-824e-3b9896f81e3c"
Sep 11 14:40:32.655395 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:32.655194    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:32.655889 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:32.655214    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:32.655889 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:32.655481    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvvkt" podUID="f2636cfe-b8a3-4c64-94f5-7026d6603aa0"
Sep 11 14:40:32.655889 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:32.655597    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsqws" podUID="1cb940cd-c21b-4118-ad56-bb823c730604"
Sep 11 14:40:32.655889 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:32.655288    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:32.655889 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:32.655718    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxv9g" podUID="3296271a-8ecd-483d-824e-3b9896f81e3c"
Sep 11 14:40:34.655396 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:34.655361    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:34.655848 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:34.655492    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxv9g" podUID="3296271a-8ecd-483d-824e-3b9896f81e3c"
Sep 11 14:40:34.655848 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:34.655363    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:34.655848 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:34.655580    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsqws" podUID="1cb940cd-c21b-4118-ad56-bb823c730604"
Sep 11 14:40:34.655848 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:34.655363    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:34.655848 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:34.655635    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvvkt" podUID="f2636cfe-b8a3-4c64-94f5-7026d6603aa0"
Sep 11 14:40:36.655921 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:36.655887    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:36.656386 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:36.655887    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:36.656386 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:36.656010    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvvkt" podUID="f2636cfe-b8a3-4c64-94f5-7026d6603aa0"
Sep 11 14:40:36.656386 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:36.656112    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsqws" podUID="1cb940cd-c21b-4118-ad56-bb823c730604"
Sep 11 14:40:36.656386 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:36.655902    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:36.656386 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:36.656219    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxv9g" podUID="3296271a-8ecd-483d-824e-3b9896f81e3c"
Sep 11 14:40:38.261895 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:38.261797    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txk6q\" (UniqueName: \"kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q\") pod \"network-check-target-qvvkt\" (UID: \"f2636cfe-b8a3-4c64-94f5-7026d6603aa0\") " pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:38.262363 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:38.261952    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:40:38.262363 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:38.261979    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:40:38.262363 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:38.261992    2568 projected.go:196] Error preparing data for projected volume kube-api-access-txk6q for pod openshift-network-diagnostics/network-check-target-qvvkt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:38.262363 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:38.262057    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q podName:f2636cfe-b8a3-4c64-94f5-7026d6603aa0 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:54.262039288 +0000 UTC m=+34.050303580 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-txk6q" (UniqueName: "kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q") pod "network-check-target-qvvkt" (UID: "f2636cfe-b8a3-4c64-94f5-7026d6603aa0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:38.363085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:38.363053    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs\") pod \"network-metrics-daemon-zsqws\" (UID: \"1cb940cd-c21b-4118-ad56-bb823c730604\") " pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:38.363253 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:38.363106    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret\") pod \"global-pull-secret-syncer-sxv9g\" (UID: \"3296271a-8ecd-483d-824e-3b9896f81e3c\") " pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:38.363253 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:38.363204    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:40:38.363253 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:38.363211    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:40:38.363397 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:38.363259    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret podName:3296271a-8ecd-483d-824e-3b9896f81e3c nodeName:}" failed. No retries permitted until 2026-09-11 14:40:54.363242703 +0000 UTC m=+34.151506995 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret") pod "global-pull-secret-syncer-sxv9g" (UID: "3296271a-8ecd-483d-824e-3b9896f81e3c") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:40:38.363397 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:38.363276    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs podName:1cb940cd-c21b-4118-ad56-bb823c730604 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:54.363268826 +0000 UTC m=+34.151533103 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs") pod "network-metrics-daemon-zsqws" (UID: "1cb940cd-c21b-4118-ad56-bb823c730604") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:40:38.658040 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:38.658006    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:38.658207 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:38.658050    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:38.658207 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:38.658124    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:38.658207 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:38.658145    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsqws" podUID="1cb940cd-c21b-4118-ad56-bb823c730604"
Sep 11 14:40:38.658332 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:38.658222    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvvkt" podUID="f2636cfe-b8a3-4c64-94f5-7026d6603aa0"
Sep 11 14:40:38.658332 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:38.658307    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxv9g" podUID="3296271a-8ecd-483d-824e-3b9896f81e3c"
Sep 11 14:40:40.656412 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:40.656378    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:40.656893 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:40.656495    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvvkt" podUID="f2636cfe-b8a3-4c64-94f5-7026d6603aa0"
Sep 11 14:40:40.656893 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:40.656651    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:40.656893 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:40.656683    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:40.656893 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:40.656782    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxv9g" podUID="3296271a-8ecd-483d-824e-3b9896f81e3c"
Sep 11 14:40:40.657066 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:40.656915    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsqws" podUID="1cb940cd-c21b-4118-ad56-bb823c730604"
Sep 11 14:40:41.847152 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:41.846710    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh" event={"ID":"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de","Type":"ContainerStarted","Data":"5766d4779dc9914e02325a2b0a609782584d424e41a4487b20ecf41edac780a2"}
Sep 11 14:40:42.659273 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.659104    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:42.659450 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.659206    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:42.659450 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:42.659367    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsqws" podUID="1cb940cd-c21b-4118-ad56-bb823c730604"
Sep 11 14:40:42.659450 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.659205    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:42.659627 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:42.659439    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxv9g" podUID="3296271a-8ecd-483d-824e-3b9896f81e3c"
Sep 11 14:40:42.659627 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:42.659532    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvvkt" podUID="f2636cfe-b8a3-4c64-94f5-7026d6603aa0"
Sep 11 14:40:42.863585 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.863551    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hhmbs" event={"ID":"e57ff2ae-a4ff-4f18-a25a-f0ee3627e106","Type":"ContainerStarted","Data":"1649bbe90bfd8854ddea8fe01313b4fa148b997b58483fa20662fb1baedd35c3"}
Sep 11 14:40:42.865067 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.865046    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xzmzt" event={"ID":"64bef023-3ded-4c9d-9e40-1b202e960fe2","Type":"ContainerStarted","Data":"c330a74811451b3d1002f16ddc128069d04a565c15317a0cd0c27a4efbca6bf2"}
Sep 11 14:40:42.866405 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.866384    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x9q88" event={"ID":"e6f17c87-b5e8-4a52-a3b4-f5b409f3e75f","Type":"ContainerStarted","Data":"c9ae47074803d57fec100132fa3d40bf200c3bf57da8ca3a6f79032723ded17a"}
Sep 11 14:40:42.867771 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.867750    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7mlmr" event={"ID":"2695284a-7a7f-42ef-9f9a-e152c61edaf3","Type":"ContainerStarted","Data":"e8756c5627d505f4cf66587a0e142e829a7f6676a6c86e09025148f80ec131c8"}
Sep 11 14:40:42.869140 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.869116    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vh5fh" event={"ID":"c974f271-da27-406b-9d10-4c1889517ce1","Type":"ContainerStarted","Data":"1a0fc6439959e3f26c810ecb14e47e0c8ee5b1d16706bfca7e47c130c1eb772d"}
Sep 11 14:40:42.870663 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.870643    2568 generic.go:356] "Generic (PLEG): container finished" podID="53f3bf9a-6da4-46bb-a51e-f0c10c0da500" containerID="9abbbae1eee20413fb551a4431f8927b3c6690174f4c630d0050c4efa170ab6a" exitCode=0
Sep 11 14:40:42.870754 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.870716    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nflgb" event={"ID":"53f3bf9a-6da4-46bb-a51e-f0c10c0da500","Type":"ContainerDied","Data":"9abbbae1eee20413fb551a4431f8927b3c6690174f4c630d0050c4efa170ab6a"}
Sep 11 14:40:42.872164 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.872093    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft" event={"ID":"4caa60a7-ed08-4ca6-a4dd-df3e16edb938","Type":"ContainerStarted","Data":"fbb4525322c7252e51b3f4b37f8c0e0e760a57032219f76bd7fd5c48880b4334"}
Sep 11 14:40:42.875082 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.875062    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh" event={"ID":"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de","Type":"ContainerStarted","Data":"8e71463d0f4ef06491acaa90aac24da55c23d4050f382eb2cc6da799b06ff7b4"}
Sep 11 14:40:42.875162 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.875117    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh" event={"ID":"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de","Type":"ContainerStarted","Data":"8944cbbf9e715c3473d2ceac546845625d27092646c390266355b48228fa44cf"}
Sep 11 14:40:42.875162 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.875147    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh" event={"ID":"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de","Type":"ContainerStarted","Data":"c944dcd19b580a607c90c0dbd9a3dbad9bdd0a4ce5f532ac5b17148266c353d0"}
Sep 11 14:40:42.875230 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.875179    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh" event={"ID":"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de","Type":"ContainerStarted","Data":"a4e517608f330799a8d3352721316b7bdc3f6ed4878f8d092715a032c6bff331"}
Sep 11 14:40:42.875230 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.875208    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh" event={"ID":"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de","Type":"ContainerStarted","Data":"de54b8cf398b1dd713aef5639e9a5336d10a28c240af7cbf2ccb7147a3c643ad"}
Sep 11 14:40:42.883296 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.883263    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-126.ec2.internal" podStartSLOduration=20.883252338 podStartE2EDuration="20.883252338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:25.819181918 +0000 UTC m=+5.607446219" watchObservedRunningTime="2026-09-11 14:40:42.883252338 +0000 UTC m=+22.671516637"
Sep 11 14:40:42.883389 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.883374    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hhmbs" podStartSLOduration=4.981616618 podStartE2EDuration="22.883371265s" totalImagesPullingTime="17.901754647s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:42.883024358 +0000 UTC m=+22.671288658" watchObservedRunningTime="2026-09-11 14:40:42.883371265 +0000 UTC m=+22.671635563"
Sep 11 14:40:42.898788 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.898753    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x9q88" podStartSLOduration=8.92426132 podStartE2EDuration="22.898741915s" totalImagesPullingTime="13.974480595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:42.898313417 +0000 UTC m=+22.686577718" watchObservedRunningTime="2026-09-11 14:40:42.898741915 +0000 UTC m=+22.687006215"
Sep 11 14:40:42.913696 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.913622    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7mlmr" podStartSLOduration=5.071889609 podStartE2EDuration="22.913604841s" totalImagesPullingTime="17.841715232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:42.913184213 +0000 UTC m=+22.701448512" watchObservedRunningTime="2026-09-11 14:40:42.913604841 +0000 UTC m=+22.701869141"
Sep 11 14:40:42.929215 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.929171    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xzmzt" podStartSLOduration=5.087767377 podStartE2EDuration="22.929154162s" totalImagesPullingTime="17.841386785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:42.929016774 +0000 UTC m=+22.717281072" watchObservedRunningTime="2026-09-11 14:40:42.929154162 +0000 UTC m=+22.717418465"
Sep 11 14:40:42.975336 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:42.975292    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vh5fh" podStartSLOduration=5.119372128 podStartE2EDuration="22.975266305s" totalImagesPullingTime="17.855894177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:42.974878696 +0000 UTC m=+22.763143013" watchObservedRunningTime="2026-09-11 14:40:42.975266305 +0000 UTC m=+22.763530606"
Sep 11 14:40:43.035310 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:43.035281    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x9q88"
Sep 11 14:40:43.036243 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:43.036217    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x9q88"
Sep 11 14:40:43.423202 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:43.423015    2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 14:40:43.599069 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:43.598921    2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T14:40:43.42319937Z","UUID":"305a9800-715b-4f20-bbed-33cfdb2cb5b8","Handler":null,"Name":"","Endpoint":""}
Sep 11 14:40:43.600250 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:43.600232    2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 14:40:43.600348 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:43.600258    2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 14:40:43.879314 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:43.879230    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pvpvd" event={"ID":"b452c8a8-b01e-4250-bc67-2708e22918bb","Type":"ContainerStarted","Data":"63092107cd24e36a60856381ebb1a5993dc871e90e5fabed730d6674f5db754a"}
Sep 11 14:40:43.882047 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:43.881904    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft" event={"ID":"4caa60a7-ed08-4ca6-a4dd-df3e16edb938","Type":"ContainerStarted","Data":"83dbe82784111e04314483892453e7748ed2c709d3e305c540d9ade550f4bbe8"}
Sep 11 14:40:43.882486 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:43.882462    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x9q88"
Sep 11 14:40:43.883219 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:43.883183    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x9q88"
Sep 11 14:40:43.898631 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:43.898577    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pvpvd" podStartSLOduration=6.062334685 podStartE2EDuration="23.898564569s" totalImagesPullingTime="17.836229884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:43.898205089 +0000 UTC m=+23.686469391" watchObservedRunningTime="2026-09-11 14:40:43.898564569 +0000 UTC m=+23.686828868"
Sep 11 14:40:44.655371 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:44.655341    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:44.655494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:44.655342    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:44.655559 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:44.655487    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsqws" podUID="1cb940cd-c21b-4118-ad56-bb823c730604"
Sep 11 14:40:44.655559 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:44.655498    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:44.655665 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:44.655585    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvvkt" podUID="f2636cfe-b8a3-4c64-94f5-7026d6603aa0"
Sep 11 14:40:44.655767 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:44.655676    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxv9g" podUID="3296271a-8ecd-483d-824e-3b9896f81e3c"
Sep 11 14:40:44.887438 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:44.887406    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft" event={"ID":"4caa60a7-ed08-4ca6-a4dd-df3e16edb938","Type":"ContainerStarted","Data":"37726da50a4cb60f9019e17859794693994a5698083022456004c5451e001641"}
Sep 11 14:40:44.890716 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:44.890692    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh" event={"ID":"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de","Type":"ContainerStarted","Data":"c5db5db62a345b8bcee5df2758f8774752ceff6ce3df8b1898731cf1ba7e4ff6"}
Sep 11 14:40:44.910960 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:44.910885    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qg9ft" podStartSLOduration=4.698283628 podStartE2EDuration="24.91087002s" totalImagesPullingTime="20.212586392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:44.910762568 +0000 UTC m=+24.699026867" watchObservedRunningTime="2026-09-11 14:40:44.91087002 +0000 UTC m=+24.699134325"
Sep 11 14:40:46.655244 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:46.655214    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:46.655696 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:46.655328    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:46.655696 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:46.655335    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvvkt" podUID="f2636cfe-b8a3-4c64-94f5-7026d6603aa0"
Sep 11 14:40:46.655696 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:46.655461    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsqws" podUID="1cb940cd-c21b-4118-ad56-bb823c730604"
Sep 11 14:40:46.655696 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:46.655507    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:46.655696 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:46.655553    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxv9g" podUID="3296271a-8ecd-483d-824e-3b9896f81e3c"
Sep 11 14:40:47.898962 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:47.898777    2568 generic.go:356] "Generic (PLEG): container finished" podID="53f3bf9a-6da4-46bb-a51e-f0c10c0da500" containerID="f5e9d788a1505ef51a8fd53d18c97a18d3c2ea832143bf35c84e5f1736329c5c" exitCode=0
Sep 11 14:40:47.899536 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:47.898848    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nflgb" event={"ID":"53f3bf9a-6da4-46bb-a51e-f0c10c0da500","Type":"ContainerDied","Data":"f5e9d788a1505ef51a8fd53d18c97a18d3c2ea832143bf35c84e5f1736329c5c"}
Sep 11 14:40:47.902914 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:47.902888    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh" event={"ID":"b5b0ac85-9e4b-4ec7-afdf-2a9082f8c6de","Type":"ContainerStarted","Data":"960abfa5f5d123a72fa1de6057f9c08ebc45e65dc54cee10c6d280facabc350e"}
Sep 11 14:40:47.903158 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:47.903142    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:47.903216 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:47.903203    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:47.917321 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:47.917297    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:47.964015 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:47.963946    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh" podStartSLOduration=10.076221955 podStartE2EDuration="27.963934629s" totalImagesPullingTime="17.887712674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:47.962756323 +0000 UTC m=+27.751020621" watchObservedRunningTime="2026-09-11 14:40:47.963934629 +0000 UTC m=+27.752198927"
Sep 11 14:40:48.658602 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:48.658570    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:48.658763 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:48.658702    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxv9g" podUID="3296271a-8ecd-483d-824e-3b9896f81e3c"
Sep 11 14:40:48.659142 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:48.659118    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:48.659281 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:48.659220    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvvkt" podUID="f2636cfe-b8a3-4c64-94f5-7026d6603aa0"
Sep 11 14:40:48.659281 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:48.659275    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:48.659392 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:48.659356    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsqws" podUID="1cb940cd-c21b-4118-ad56-bb823c730604"
Sep 11 14:40:48.905503 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:48.905472    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:48.919658 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:48.919603    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:40:49.356332 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:49.356092    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sxv9g"]
Sep 11 14:40:49.356446 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:49.356437    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:49.356550 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:49.356526    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxv9g" podUID="3296271a-8ecd-483d-824e-3b9896f81e3c"
Sep 11 14:40:49.359153 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:49.359121    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qvvkt"]
Sep 11 14:40:49.359258 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:49.359220    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:49.359337 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:49.359308    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvvkt" podUID="f2636cfe-b8a3-4c64-94f5-7026d6603aa0"
Sep 11 14:40:49.359754 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:49.359725    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zsqws"]
Sep 11 14:40:49.359850 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:49.359841    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:49.359968 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:49.359947    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsqws" podUID="1cb940cd-c21b-4118-ad56-bb823c730604"
Sep 11 14:40:49.909045 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:49.909010    2568 generic.go:356] "Generic (PLEG): container finished" podID="53f3bf9a-6da4-46bb-a51e-f0c10c0da500" containerID="2f045764a927320a44b1b3e37049dd25c2c53e2e7d7fd11d36107a31c64a0c7e" exitCode=0
Sep 11 14:40:49.909506 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:49.909075    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nflgb" event={"ID":"53f3bf9a-6da4-46bb-a51e-f0c10c0da500","Type":"ContainerDied","Data":"2f045764a927320a44b1b3e37049dd25c2c53e2e7d7fd11d36107a31c64a0c7e"}
Sep 11 14:40:50.656682 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:50.656659    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:50.656844 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:50.656746    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvvkt" podUID="f2636cfe-b8a3-4c64-94f5-7026d6603aa0"
Sep 11 14:40:50.656844 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:50.656833    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:50.656957 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:50.656855    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:50.657006 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:50.656969    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsqws" podUID="1cb940cd-c21b-4118-ad56-bb823c730604"
Sep 11 14:40:50.657102 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:50.657040    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxv9g" podUID="3296271a-8ecd-483d-824e-3b9896f81e3c"
Sep 11 14:40:51.915652 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:51.915568    2568 generic.go:356] "Generic (PLEG): container finished" podID="53f3bf9a-6da4-46bb-a51e-f0c10c0da500" containerID="880306e07c90fd1b26d09481f4ac8eb923babfeee719e63693dec445cb33fc45" exitCode=0
Sep 11 14:40:51.915652 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:51.915619    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nflgb" event={"ID":"53f3bf9a-6da4-46bb-a51e-f0c10c0da500","Type":"ContainerDied","Data":"880306e07c90fd1b26d09481f4ac8eb923babfeee719e63693dec445cb33fc45"}
Sep 11 14:40:52.655535 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:52.655500    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:52.655535 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:52.655517    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:52.655720 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:52.655601    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxv9g" podUID="3296271a-8ecd-483d-824e-3b9896f81e3c"
Sep 11 14:40:52.655720 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:52.655649    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zsqws" podUID="1cb940cd-c21b-4118-ad56-bb823c730604"
Sep 11 14:40:52.655720 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:52.655685    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:52.655842 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:52.655771    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvvkt" podUID="f2636cfe-b8a3-4c64-94f5-7026d6603aa0"
Sep 11 14:40:54.277572 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.277542    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txk6q\" (UniqueName: \"kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q\") pod \"network-check-target-qvvkt\" (UID: \"f2636cfe-b8a3-4c64-94f5-7026d6603aa0\") " pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:54.277942 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:54.277737    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:40:54.277942 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:54.277762    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:40:54.277942 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:54.277776    2568 projected.go:196] Error preparing data for projected volume kube-api-access-txk6q for pod openshift-network-diagnostics/network-check-target-qvvkt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:54.277942 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:54.277851    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q podName:f2636cfe-b8a3-4c64-94f5-7026d6603aa0 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:26.277834642 +0000 UTC m=+66.066098938 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-txk6q" (UniqueName: "kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q") pod "network-check-target-qvvkt" (UID: "f2636cfe-b8a3-4c64-94f5-7026d6603aa0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:40:54.378242 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.378210    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret\") pod \"global-pull-secret-syncer-sxv9g\" (UID: \"3296271a-8ecd-483d-824e-3b9896f81e3c\") " pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:54.378448 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.378260    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs\") pod \"network-metrics-daemon-zsqws\" (UID: \"1cb940cd-c21b-4118-ad56-bb823c730604\") " pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:54.378448 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:54.378383    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:40:54.378567 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:54.378454    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs podName:1cb940cd-c21b-4118-ad56-bb823c730604 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:26.378428419 +0000 UTC m=+66.166692715 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs") pod "network-metrics-daemon-zsqws" (UID: "1cb940cd-c21b-4118-ad56-bb823c730604") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:40:54.378567 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:54.378383    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:40:54.378567 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:54.378541    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret podName:3296271a-8ecd-483d-824e-3b9896f81e3c nodeName:}" failed. No retries permitted until 2026-09-11 14:41:26.378521252 +0000 UTC m=+66.166785544 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret") pod "global-pull-secret-syncer-sxv9g" (UID: "3296271a-8ecd-483d-824e-3b9896f81e3c") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:40:54.449977 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.449948    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-126.ec2.internal" event="NodeReady"
Sep 11 14:40:54.450188 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.450120    2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 14:40:54.500473 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.500408    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x6c4c"]
Sep 11 14:40:54.517988 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.517961    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-72l5p"]
Sep 11 14:40:54.518146 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.518081    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:40:54.521263 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.521242    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 14:40:54.521361 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.521266    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 14:40:54.521361 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.521293    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vx9kl\""
Sep 11 14:40:54.534676 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.534637    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x6c4c"]
Sep 11 14:40:54.534787 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.534726    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-72l5p"
Sep 11 14:40:54.534787 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.534768    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-72l5p"]
Sep 11 14:40:54.538738 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.538520    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 14:40:54.538904 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.538872    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 14:40:54.539200 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.539166    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tl5m6\""
Sep 11 14:40:54.539200 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.539184    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 14:40:54.655990 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.655962    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:40:54.656178 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.655962    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:40:54.656256 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.655962    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:40:54.661127 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.661105    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 14:40:54.661321 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.661302    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 14:40:54.661397 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.661331    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jjd7d\""
Sep 11 14:40:54.661672 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.661651    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 14:40:54.661767 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.661746    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xdw2d\""
Sep 11 14:40:54.661844 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.661827    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 14:40:54.679891 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.679871    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5e7d1c8-7b1d-494f-9c76-fea7135dd88c-cert\") pod \"ingress-canary-72l5p\" (UID: \"d5e7d1c8-7b1d-494f-9c76-fea7135dd88c\") " pod="openshift-ingress-canary/ingress-canary-72l5p"
Sep 11 14:40:54.680010 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.679922    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-metrics-tls\") pod \"dns-default-x6c4c\" (UID: \"a0d57bd1-aa8e-41b5-b3a2-720c733285f1\") " pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:40:54.680010 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.679951    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-config-volume\") pod \"dns-default-x6c4c\" (UID: \"a0d57bd1-aa8e-41b5-b3a2-720c733285f1\") " pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:40:54.680010 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.679977    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrzzt\" (UniqueName: \"kubernetes.io/projected/d5e7d1c8-7b1d-494f-9c76-fea7135dd88c-kube-api-access-rrzzt\") pod \"ingress-canary-72l5p\" (UID: \"d5e7d1c8-7b1d-494f-9c76-fea7135dd88c\") " pod="openshift-ingress-canary/ingress-canary-72l5p"
Sep 11 14:40:54.680166 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.680047    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl76z\" (UniqueName: \"kubernetes.io/projected/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-kube-api-access-tl76z\") pod \"dns-default-x6c4c\" (UID: \"a0d57bd1-aa8e-41b5-b3a2-720c733285f1\") " pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:40:54.680166 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.680078    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-tmp-dir\") pod \"dns-default-x6c4c\" (UID: \"a0d57bd1-aa8e-41b5-b3a2-720c733285f1\") " pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:40:54.781278 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.781245    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl76z\" (UniqueName: \"kubernetes.io/projected/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-kube-api-access-tl76z\") pod \"dns-default-x6c4c\" (UID: \"a0d57bd1-aa8e-41b5-b3a2-720c733285f1\") " pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:40:54.781462 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.781294    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-tmp-dir\") pod \"dns-default-x6c4c\" (UID: \"a0d57bd1-aa8e-41b5-b3a2-720c733285f1\") " pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:40:54.781462 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.781426    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5e7d1c8-7b1d-494f-9c76-fea7135dd88c-cert\") pod \"ingress-canary-72l5p\" (UID: \"d5e7d1c8-7b1d-494f-9c76-fea7135dd88c\") " pod="openshift-ingress-canary/ingress-canary-72l5p"
Sep 11 14:40:54.781586 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.781495    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-metrics-tls\") pod \"dns-default-x6c4c\" (UID: \"a0d57bd1-aa8e-41b5-b3a2-720c733285f1\") " pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:40:54.781586 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.781544    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-config-volume\") pod \"dns-default-x6c4c\" (UID: \"a0d57bd1-aa8e-41b5-b3a2-720c733285f1\") " pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:40:54.781586 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.781569    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrzzt\" (UniqueName: \"kubernetes.io/projected/d5e7d1c8-7b1d-494f-9c76-fea7135dd88c-kube-api-access-rrzzt\") pod \"ingress-canary-72l5p\" (UID: \"d5e7d1c8-7b1d-494f-9c76-fea7135dd88c\") " pod="openshift-ingress-canary/ingress-canary-72l5p"
Sep 11 14:40:54.781733 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.781634    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-tmp-dir\") pod \"dns-default-x6c4c\" (UID: \"a0d57bd1-aa8e-41b5-b3a2-720c733285f1\") " pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:40:54.781890 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:54.781871    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 14:40:54.781983 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:54.781947    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-metrics-tls podName:a0d57bd1-aa8e-41b5-b3a2-720c733285f1 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:55.281928103 +0000 UTC m=+35.070192387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-metrics-tls") pod "dns-default-x6c4c" (UID: "a0d57bd1-aa8e-41b5-b3a2-720c733285f1") : secret "dns-default-metrics-tls" not found
Sep 11 14:40:54.782229 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.782208    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-config-volume\") pod \"dns-default-x6c4c\" (UID: \"a0d57bd1-aa8e-41b5-b3a2-720c733285f1\") " pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:40:54.786214 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.786190    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5e7d1c8-7b1d-494f-9c76-fea7135dd88c-cert\") pod \"ingress-canary-72l5p\" (UID: \"d5e7d1c8-7b1d-494f-9c76-fea7135dd88c\") " pod="openshift-ingress-canary/ingress-canary-72l5p"
Sep 11 14:40:54.792510 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.792489    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl76z\" (UniqueName: \"kubernetes.io/projected/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-kube-api-access-tl76z\") pod \"dns-default-x6c4c\" (UID: \"a0d57bd1-aa8e-41b5-b3a2-720c733285f1\") " pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:40:54.792617 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.792495    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrzzt\" (UniqueName: \"kubernetes.io/projected/d5e7d1c8-7b1d-494f-9c76-fea7135dd88c-kube-api-access-rrzzt\") pod \"ingress-canary-72l5p\" (UID: \"d5e7d1c8-7b1d-494f-9c76-fea7135dd88c\") " pod="openshift-ingress-canary/ingress-canary-72l5p"
Sep 11 14:40:54.844553 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:54.844531    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-72l5p"
Sep 11 14:40:55.158877 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:55.158593    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-72l5p"]
Sep 11 14:40:55.286519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:55.286484    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-metrics-tls\") pod \"dns-default-x6c4c\" (UID: \"a0d57bd1-aa8e-41b5-b3a2-720c733285f1\") " pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:40:55.286916 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:55.286588    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 14:40:55.286916 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:55.286650    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-metrics-tls podName:a0d57bd1-aa8e-41b5-b3a2-720c733285f1 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:56.286635049 +0000 UTC m=+36.074899351 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-metrics-tls") pod "dns-default-x6c4c" (UID: "a0d57bd1-aa8e-41b5-b3a2-720c733285f1") : secret "dns-default-metrics-tls" not found
Sep 11 14:40:55.989170 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:55.989141    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zhgrn"]
Sep 11 14:40:56.019577 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.019545    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zhgrn"]
Sep 11 14:40:56.019738 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.019644    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zhgrn"
Sep 11 14:40:56.023464 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.023436    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 11 14:40:56.023594 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.023436    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 11 14:40:56.024603 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.024581    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xpkft\""
Sep 11 14:40:56.092775 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.092743    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jkfc\" (UniqueName: \"kubernetes.io/projected/0adfdfe5-37aa-4eac-b3fe-7d9de1053786-kube-api-access-8jkfc\") pod \"volume-data-source-validator-86695b7cc7-zhgrn\" (UID: \"0adfdfe5-37aa-4eac-b3fe-7d9de1053786\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zhgrn"
Sep 11 14:40:56.155401 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.155374    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-rbgf9"]
Sep 11 14:40:56.176885 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.176856    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6dbb8d47db-qzqwf"]
Sep 11 14:40:56.177043 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.176977    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rbgf9"
Sep 11 14:40:56.182490 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.182463    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 11 14:40:56.182621 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.182587    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 11 14:40:56.182689 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.182629    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 11 14:40:56.182751 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.182473    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bcl5b\""
Sep 11 14:40:56.183050 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.183034    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 11 14:40:56.192935 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.192901    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-rbgf9"]
Sep 11 14:40:56.192935 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.192948    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6dbb8d47db-qzqwf"]
Sep 11 14:40:56.193111 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.193034    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:40:56.193196 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.193175    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jkfc\" (UniqueName: \"kubernetes.io/projected/0adfdfe5-37aa-4eac-b3fe-7d9de1053786-kube-api-access-8jkfc\") pod \"volume-data-source-validator-86695b7cc7-zhgrn\" (UID: \"0adfdfe5-37aa-4eac-b3fe-7d9de1053786\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zhgrn"
Sep 11 14:40:56.196279 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.196260    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 11 14:40:56.196805 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.196788    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 14:40:56.196915 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.196837    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 11 14:40:56.197354 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.197201    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4dmnm\""
Sep 11 14:40:56.197354 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.197296    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 14:40:56.197354 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.197299    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 11 14:40:56.197354 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.197320    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 11 14:40:56.204855 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.204833    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jkfc\" (UniqueName: \"kubernetes.io/projected/0adfdfe5-37aa-4eac-b3fe-7d9de1053786-kube-api-access-8jkfc\") pod \"volume-data-source-validator-86695b7cc7-zhgrn\" (UID: \"0adfdfe5-37aa-4eac-b3fe-7d9de1053786\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zhgrn"
Sep 11 14:40:56.294465 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.294428    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-default-certificate\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:40:56.295026 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.294510    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-metrics-tls\") pod \"dns-default-x6c4c\" (UID: \"a0d57bd1-aa8e-41b5-b3a2-720c733285f1\") " pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:40:56.295026 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.294542    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a12f3fa6-b08b-436f-bef3-ebe50265cc52-serving-cert\") pod \"service-ca-operator-cf7c776b7-rbgf9\" (UID: \"a12f3fa6-b08b-436f-bef3-ebe50265cc52\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rbgf9"
Sep 11 14:40:56.295026 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.294559    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:40:56.295026 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.294585    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-metrics-certs\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:40:56.295026 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:56.294626    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 14:40:56.295026 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.294645    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-stats-auth\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:40:56.295026 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.294686    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a12f3fa6-b08b-436f-bef3-ebe50265cc52-config\") pod \"service-ca-operator-cf7c776b7-rbgf9\" (UID: \"a12f3fa6-b08b-436f-bef3-ebe50265cc52\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rbgf9"
Sep 11 14:40:56.295026 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:56.294700    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-metrics-tls podName:a0d57bd1-aa8e-41b5-b3a2-720c733285f1 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:58.294681382 +0000 UTC m=+38.082945679 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-metrics-tls") pod "dns-default-x6c4c" (UID: "a0d57bd1-aa8e-41b5-b3a2-720c733285f1") : secret "dns-default-metrics-tls" not found
Sep 11 14:40:56.295026 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.294742    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29q6t\" (UniqueName: \"kubernetes.io/projected/a12f3fa6-b08b-436f-bef3-ebe50265cc52-kube-api-access-29q6t\") pod \"service-ca-operator-cf7c776b7-rbgf9\" (UID: \"a12f3fa6-b08b-436f-bef3-ebe50265cc52\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rbgf9"
Sep 11 14:40:56.295026 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.294769    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wdn2\" (UniqueName: \"kubernetes.io/projected/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-kube-api-access-7wdn2\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:40:56.330186 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.330161    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zhgrn"
Sep 11 14:40:56.396085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.396053    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-default-certificate\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:40:56.396233 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.396117    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a12f3fa6-b08b-436f-bef3-ebe50265cc52-serving-cert\") pod \"service-ca-operator-cf7c776b7-rbgf9\" (UID: \"a12f3fa6-b08b-436f-bef3-ebe50265cc52\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rbgf9"
Sep 11 14:40:56.396233 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.396140    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:40:56.396233 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.396163    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-metrics-certs\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:40:56.396233 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.396196    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-stats-auth\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:40:56.396506 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:56.396300    2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 14:40:56.396506 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:56.396369    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-metrics-certs podName:80c64211-15e9-4ac3-8660-cb0ba7ae9b90 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:56.896349242 +0000 UTC m=+36.684613518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-metrics-certs") pod "router-default-6dbb8d47db-qzqwf" (UID: "80c64211-15e9-4ac3-8660-cb0ba7ae9b90") : secret "router-metrics-certs-default" not found
Sep 11 14:40:56.396506 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:56.396389    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle podName:80c64211-15e9-4ac3-8660-cb0ba7ae9b90 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:56.896379482 +0000 UTC m=+36.684643761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle") pod "router-default-6dbb8d47db-qzqwf" (UID: "80c64211-15e9-4ac3-8660-cb0ba7ae9b90") : configmap references non-existent config key: service-ca.crt
Sep 11 14:40:56.396506 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.396422    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a12f3fa6-b08b-436f-bef3-ebe50265cc52-config\") pod \"service-ca-operator-cf7c776b7-rbgf9\" (UID: \"a12f3fa6-b08b-436f-bef3-ebe50265cc52\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rbgf9"
Sep 11 14:40:56.396506 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.396452    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29q6t\" (UniqueName: \"kubernetes.io/projected/a12f3fa6-b08b-436f-bef3-ebe50265cc52-kube-api-access-29q6t\") pod \"service-ca-operator-cf7c776b7-rbgf9\" (UID: \"a12f3fa6-b08b-436f-bef3-ebe50265cc52\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rbgf9"
Sep 11 14:40:56.396769 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.396573    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wdn2\" (UniqueName: \"kubernetes.io/projected/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-kube-api-access-7wdn2\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:40:56.397044 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.397021    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a12f3fa6-b08b-436f-bef3-ebe50265cc52-config\") pod \"service-ca-operator-cf7c776b7-rbgf9\" (UID: \"a12f3fa6-b08b-436f-bef3-ebe50265cc52\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rbgf9"
Sep 11 14:40:56.399077 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.399031    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a12f3fa6-b08b-436f-bef3-ebe50265cc52-serving-cert\") pod \"service-ca-operator-cf7c776b7-rbgf9\" (UID: \"a12f3fa6-b08b-436f-bef3-ebe50265cc52\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rbgf9"
Sep 11 14:40:56.399184 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.399133    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-stats-auth\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:40:56.399245 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.399200    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-default-certificate\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:40:56.411792 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.411760    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wdn2\" (UniqueName: \"kubernetes.io/projected/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-kube-api-access-7wdn2\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:40:56.412937 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.412916    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29q6t\" (UniqueName: \"kubernetes.io/projected/a12f3fa6-b08b-436f-bef3-ebe50265cc52-kube-api-access-29q6t\") pod \"service-ca-operator-cf7c776b7-rbgf9\" (UID: \"a12f3fa6-b08b-436f-bef3-ebe50265cc52\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rbgf9"
Sep 11 14:40:56.490373 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.490342    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rbgf9"
Sep 11 14:40:56.541156 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:56.541125    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e7d1c8_7b1d_494f_9c76_fea7135dd88c.slice/crio-1d41738e2c35c6ede767a12e610581886448d866293b5f9468ca2f470bf464a1 WatchSource:0}: Error finding container 1d41738e2c35c6ede767a12e610581886448d866293b5f9468ca2f470bf464a1: Status 404 returned error can't find the container with id 1d41738e2c35c6ede767a12e610581886448d866293b5f9468ca2f470bf464a1
Sep 11 14:40:56.810263 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.810190    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-rbgf9"]
Sep 11 14:40:56.813335 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.813308    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zhgrn"]
Sep 11 14:40:56.864305 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:56.864270    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12f3fa6_b08b_436f_bef3_ebe50265cc52.slice/crio-d2de5b7066018dc520015e507ce82eb74e3bd95532773cf2de64c2d1bf4b39c0 WatchSource:0}: Error finding container d2de5b7066018dc520015e507ce82eb74e3bd95532773cf2de64c2d1bf4b39c0: Status 404 returned error can't find the container with id d2de5b7066018dc520015e507ce82eb74e3bd95532773cf2de64c2d1bf4b39c0
Sep 11 14:40:56.864980 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:40:56.864892    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0adfdfe5_37aa_4eac_b3fe_7d9de1053786.slice/crio-a6bde0db4fb91b443e9e39ecc3d76a2d646209948c46f15b09019919f2ed20e2 WatchSource:0}: Error finding container a6bde0db4fb91b443e9e39ecc3d76a2d646209948c46f15b09019919f2ed20e2: Status 404 returned error can't find the container with id a6bde0db4fb91b443e9e39ecc3d76a2d646209948c46f15b09019919f2ed20e2
Sep 11 14:40:56.901348 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.901321    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:40:56.901454 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.901365    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-metrics-certs\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:40:56.901513 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:56.901478    2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 14:40:56.901513 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:56.901496    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle podName:80c64211-15e9-4ac3-8660-cb0ba7ae9b90 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:57.901470331 +0000 UTC m=+37.689734610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle") pod "router-default-6dbb8d47db-qzqwf" (UID: "80c64211-15e9-4ac3-8660-cb0ba7ae9b90") : configmap references non-existent config key: service-ca.crt
Sep 11 14:40:56.901591 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:56.901527    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-metrics-certs podName:80c64211-15e9-4ac3-8660-cb0ba7ae9b90 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:57.901510531 +0000 UTC m=+37.689774810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-metrics-certs") pod "router-default-6dbb8d47db-qzqwf" (UID: "80c64211-15e9-4ac3-8660-cb0ba7ae9b90") : secret "router-metrics-certs-default" not found
Sep 11 14:40:56.929867 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.929840    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-72l5p" event={"ID":"d5e7d1c8-7b1d-494f-9c76-fea7135dd88c","Type":"ContainerStarted","Data":"1d41738e2c35c6ede767a12e610581886448d866293b5f9468ca2f470bf464a1"}
Sep 11 14:40:56.930938 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.930912    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rbgf9" event={"ID":"a12f3fa6-b08b-436f-bef3-ebe50265cc52","Type":"ContainerStarted","Data":"d2de5b7066018dc520015e507ce82eb74e3bd95532773cf2de64c2d1bf4b39c0"}
Sep 11 14:40:56.932038 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:56.932018    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zhgrn" event={"ID":"0adfdfe5-37aa-4eac-b3fe-7d9de1053786","Type":"ContainerStarted","Data":"a6bde0db4fb91b443e9e39ecc3d76a2d646209948c46f15b09019919f2ed20e2"}
Sep 11 14:40:57.909959 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:57.909923    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:40:57.909959 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:57.909972    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-metrics-certs\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:40:57.910787 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:57.910094    2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 14:40:57.910787 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:57.910156    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-metrics-certs podName:80c64211-15e9-4ac3-8660-cb0ba7ae9b90 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:59.910137601 +0000 UTC m=+39.698401901 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-metrics-certs") pod "router-default-6dbb8d47db-qzqwf" (UID: "80c64211-15e9-4ac3-8660-cb0ba7ae9b90") : secret "router-metrics-certs-default" not found
Sep 11 14:40:57.910787 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:57.910241    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle podName:80c64211-15e9-4ac3-8660-cb0ba7ae9b90 nodeName:}" failed. No retries permitted until 2026-09-11 14:40:59.910221029 +0000 UTC m=+39.698485327 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle") pod "router-default-6dbb8d47db-qzqwf" (UID: "80c64211-15e9-4ac3-8660-cb0ba7ae9b90") : configmap references non-existent config key: service-ca.crt
Sep 11 14:40:57.938777 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:57.938741    2568 generic.go:356] "Generic (PLEG): container finished" podID="53f3bf9a-6da4-46bb-a51e-f0c10c0da500" containerID="4f6c083c51ae0dc033acfa4d77233bfdeed8c49aab00b039ae0c0dfd45e0ddc1" exitCode=0
Sep 11 14:40:57.938955 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:57.938797    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nflgb" event={"ID":"53f3bf9a-6da4-46bb-a51e-f0c10c0da500","Type":"ContainerDied","Data":"4f6c083c51ae0dc033acfa4d77233bfdeed8c49aab00b039ae0c0dfd45e0ddc1"}
Sep 11 14:40:58.312714 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:58.312681    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-metrics-tls\") pod \"dns-default-x6c4c\" (UID: \"a0d57bd1-aa8e-41b5-b3a2-720c733285f1\") " pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:40:58.312908 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:58.312883    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 14:40:58.312978 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:58.312965    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-metrics-tls podName:a0d57bd1-aa8e-41b5-b3a2-720c733285f1 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:02.312940305 +0000 UTC m=+42.101204585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-metrics-tls") pod "dns-default-x6c4c" (UID: "a0d57bd1-aa8e-41b5-b3a2-720c733285f1") : secret "dns-default-metrics-tls" not found
Sep 11 14:40:59.927408 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:59.927370    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:40:59.927408 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:40:59.927410    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-metrics-certs\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:40:59.927757 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:59.927497    2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 14:40:59.927757 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:59.927528    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle podName:80c64211-15e9-4ac3-8660-cb0ba7ae9b90 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:03.92750959 +0000 UTC m=+43.715773882 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle") pod "router-default-6dbb8d47db-qzqwf" (UID: "80c64211-15e9-4ac3-8660-cb0ba7ae9b90") : configmap references non-existent config key: service-ca.crt
Sep 11 14:40:59.927757 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:40:59.927551    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-metrics-certs podName:80c64211-15e9-4ac3-8660-cb0ba7ae9b90 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:03.927544267 +0000 UTC m=+43.715808544 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-metrics-certs") pod "router-default-6dbb8d47db-qzqwf" (UID: "80c64211-15e9-4ac3-8660-cb0ba7ae9b90") : secret "router-metrics-certs-default" not found
Sep 11 14:41:00.947691 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:00.947647    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zhgrn" event={"ID":"0adfdfe5-37aa-4eac-b3fe-7d9de1053786","Type":"ContainerStarted","Data":"1af376117a94c2406ab0f5d6c20b30071c100a163975c588297fe7fb28700f8b"}
Sep 11 14:41:00.948965 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:00.948938    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s5plj"]
Sep 11 14:41:00.956360 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:00.955799    2568 generic.go:356] "Generic (PLEG): container finished" podID="53f3bf9a-6da4-46bb-a51e-f0c10c0da500" containerID="678ac71ccd171d09761bc240c899b7faa2ed68cd275788abf32c730e466a1bc1" exitCode=0
Sep 11 14:41:00.966362 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:00.966343    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s5plj"]
Sep 11 14:41:00.966458 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:00.966380    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-72l5p" event={"ID":"d5e7d1c8-7b1d-494f-9c76-fea7135dd88c","Type":"ContainerStarted","Data":"81d8beb0f8ebe022d0d5ada74db6c6a9fbbc7cb09c3899bbebf4c2da4987a662"}
Sep 11 14:41:00.966529 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:00.966466    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:00.966529 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:00.966479    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rbgf9" event={"ID":"a12f3fa6-b08b-436f-bef3-ebe50265cc52","Type":"ContainerStarted","Data":"5f2919bb55b98631148b0a8d588f5b3fbf1c3c05f08b9a8bd5778a2bc723b2cd"}
Sep 11 14:41:00.966529 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:00.966494    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nflgb" event={"ID":"53f3bf9a-6da4-46bb-a51e-f0c10c0da500","Type":"ContainerDied","Data":"678ac71ccd171d09761bc240c899b7faa2ed68cd275788abf32c730e466a1bc1"}
Sep 11 14:41:00.969032 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:00.968981    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 14:41:00.969032 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:00.968994    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 14:41:00.969200 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:00.969051    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6tvzn\""
Sep 11 14:41:00.969200 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:00.968995    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 14:41:00.969200 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:00.969118    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 14:41:00.971896 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:00.971841    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-zhgrn" podStartSLOduration=2.8991857530000003 podStartE2EDuration="5.971804016s" totalImagesPullingTime="3.072618263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:00.971787967 +0000 UTC m=+40.760052266" watchObservedRunningTime="2026-09-11 14:41:00.971804016 +0000 UTC m=+40.760068316"
Sep 11 14:41:01.028049 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:01.028004    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rbgf9" podStartSLOduration=1.9490156939999999 podStartE2EDuration="5.027988248s" totalImagesPullingTime="3.078972554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:00.995412393 +0000 UTC m=+40.783676692" watchObservedRunningTime="2026-09-11 14:41:01.027988248 +0000 UTC m=+40.816252548"
Sep 11 14:41:01.084135 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:01.084086    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-72l5p" podStartSLOduration=3.667197995 podStartE2EDuration="7.084069893s" totalImagesPullingTime="3.416871898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:01.08328191 +0000 UTC m=+40.871546209" watchObservedRunningTime="2026-09-11 14:41:01.084069893 +0000 UTC m=+40.872334192"
Sep 11 14:41:01.138637 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:01.138604    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69s5t\" (UniqueName: \"kubernetes.io/projected/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-api-access-69s5t\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:01.138637 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:01.138640    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8a46ee3-bfd6-45df-b4f3-191914102098-crio-socket\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:01.138883 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:01.138661    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8a46ee3-bfd6-45df-b4f3-191914102098-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:01.138883 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:01.138835    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8a46ee3-bfd6-45df-b4f3-191914102098-data-volume\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:01.138995 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:01.138933    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:01.239929 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:01.239849    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69s5t\" (UniqueName: \"kubernetes.io/projected/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-api-access-69s5t\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:01.239929 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:01.239890    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8a46ee3-bfd6-45df-b4f3-191914102098-crio-socket\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:01.239929 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:01.239907    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8a46ee3-bfd6-45df-b4f3-191914102098-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:01.239929 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:01.239938    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8a46ee3-bfd6-45df-b4f3-191914102098-data-volume\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:01.240250 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:01.239953    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:01.240250 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:01.240025    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 14:41:01.240250 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:01.240048    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 14:41:01.240250 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:01.240103    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-rbac-proxy-cm podName:d8a46ee3-bfd6-45df-b4f3-191914102098 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:01.740086943 +0000 UTC m=+41.528351233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s5plj" (UID: "d8a46ee3-bfd6-45df-b4f3-191914102098") : configmap "kube-rbac-proxy" not found
Sep 11 14:41:01.240250 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:01.240121    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8a46ee3-bfd6-45df-b4f3-191914102098-crio-socket\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:01.240250 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:01.240196    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8a46ee3-bfd6-45df-b4f3-191914102098-insights-runtime-extractor-tls podName:d8a46ee3-bfd6-45df-b4f3-191914102098 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:01.740183325 +0000 UTC m=+41.528447602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8a46ee3-bfd6-45df-b4f3-191914102098-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s5plj" (UID: "d8a46ee3-bfd6-45df-b4f3-191914102098") : secret "insights-runtime-extractor-tls" not found
Sep 11 14:41:01.240479 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:01.240255    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8a46ee3-bfd6-45df-b4f3-191914102098-data-volume\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:01.248924 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:01.248903    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69s5t\" (UniqueName: \"kubernetes.io/projected/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-api-access-69s5t\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:01.744498 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:01.744458    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8a46ee3-bfd6-45df-b4f3-191914102098-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:01.744673 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:01.744512    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:01.744673 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:01.744599    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 14:41:01.744673 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:01.744663    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8a46ee3-bfd6-45df-b4f3-191914102098-insights-runtime-extractor-tls podName:d8a46ee3-bfd6-45df-b4f3-191914102098 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:02.744649726 +0000 UTC m=+42.532914003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8a46ee3-bfd6-45df-b4f3-191914102098-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s5plj" (UID: "d8a46ee3-bfd6-45df-b4f3-191914102098") : secret "insights-runtime-extractor-tls" not found
Sep 11 14:41:01.744810 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:01.744609    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 14:41:01.744810 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:01.744732    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-rbac-proxy-cm podName:d8a46ee3-bfd6-45df-b4f3-191914102098 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:02.744720645 +0000 UTC m=+42.532984939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s5plj" (UID: "d8a46ee3-bfd6-45df-b4f3-191914102098") : configmap "kube-rbac-proxy" not found
Sep 11 14:41:01.961511 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:01.961470    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nflgb" event={"ID":"53f3bf9a-6da4-46bb-a51e-f0c10c0da500","Type":"ContainerStarted","Data":"93c8ef2fd3d039937c2d11a20460771724688261d34d6ea2c2a398ddd24a0855"}
Sep 11 14:41:01.987345 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:01.987222    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nflgb" podStartSLOduration=12.305915633 podStartE2EDuration="41.987204429s" totalImagesPullingTime="29.681288796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:01.985930527 +0000 UTC m=+41.774194825" watchObservedRunningTime="2026-09-11 14:41:01.987204429 +0000 UTC m=+41.775468727"
Sep 11 14:41:02.348801 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:02.348757    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-metrics-tls\") pod \"dns-default-x6c4c\" (UID: \"a0d57bd1-aa8e-41b5-b3a2-720c733285f1\") " pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:41:02.348962 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:02.348937    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 14:41:02.349018 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:02.349004    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-metrics-tls podName:a0d57bd1-aa8e-41b5-b3a2-720c733285f1 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:10.348988287 +0000 UTC m=+50.137252564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-metrics-tls") pod "dns-default-x6c4c" (UID: "a0d57bd1-aa8e-41b5-b3a2-720c733285f1") : secret "dns-default-metrics-tls" not found
Sep 11 14:41:02.751854 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:02.751756    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:02.752000 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:02.751889    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8a46ee3-bfd6-45df-b4f3-191914102098-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:02.752000 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:02.751896    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 14:41:02.752000 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:02.751962    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-rbac-proxy-cm podName:d8a46ee3-bfd6-45df-b4f3-191914102098 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:04.751943897 +0000 UTC m=+44.540208188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s5plj" (UID: "d8a46ee3-bfd6-45df-b4f3-191914102098") : configmap "kube-rbac-proxy" not found
Sep 11 14:41:02.752000 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:02.751995    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 14:41:02.752133 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:02.752031    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8a46ee3-bfd6-45df-b4f3-191914102098-insights-runtime-extractor-tls podName:d8a46ee3-bfd6-45df-b4f3-191914102098 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:04.752020163 +0000 UTC m=+44.540284457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8a46ee3-bfd6-45df-b4f3-191914102098-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s5plj" (UID: "d8a46ee3-bfd6-45df-b4f3-191914102098") : secret "insights-runtime-extractor-tls" not found
Sep 11 14:41:03.003345 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.003274    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xzmzt_64bef023-3ded-4c9d-9e40-1b202e960fe2/dns-node-resolver/0.log"
Sep 11 14:41:03.409389 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.409354    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-g8qdn"]
Sep 11 14:41:03.434727 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.434700    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-g8qdn"]
Sep 11 14:41:03.434881 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.434805    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-g8qdn"
Sep 11 14:41:03.437593 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.437572    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 11 14:41:03.438645 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.438622    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8jj7j\""
Sep 11 14:41:03.438645 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.438639    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 11 14:41:03.438793 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.438673    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 11 14:41:03.438793 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.438625    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 11 14:41:03.557499 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.557476    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtfv7\" (UniqueName: \"kubernetes.io/projected/3630d73a-3222-4dde-a46c-246b5cf499a7-kube-api-access-vtfv7\") pod \"service-ca-568869fdc5-g8qdn\" (UID: \"3630d73a-3222-4dde-a46c-246b5cf499a7\") " pod="openshift-service-ca/service-ca-568869fdc5-g8qdn"
Sep 11 14:41:03.557609 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.557583    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3630d73a-3222-4dde-a46c-246b5cf499a7-signing-cabundle\") pod \"service-ca-568869fdc5-g8qdn\" (UID: \"3630d73a-3222-4dde-a46c-246b5cf499a7\") " pod="openshift-service-ca/service-ca-568869fdc5-g8qdn"
Sep 11 14:41:03.557649 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.557632    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3630d73a-3222-4dde-a46c-246b5cf499a7-signing-key\") pod \"service-ca-568869fdc5-g8qdn\" (UID: \"3630d73a-3222-4dde-a46c-246b5cf499a7\") " pod="openshift-service-ca/service-ca-568869fdc5-g8qdn"
Sep 11 14:41:03.658212 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.658186    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3630d73a-3222-4dde-a46c-246b5cf499a7-signing-cabundle\") pod \"service-ca-568869fdc5-g8qdn\" (UID: \"3630d73a-3222-4dde-a46c-246b5cf499a7\") " pod="openshift-service-ca/service-ca-568869fdc5-g8qdn"
Sep 11 14:41:03.658323 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.658231    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3630d73a-3222-4dde-a46c-246b5cf499a7-signing-key\") pod \"service-ca-568869fdc5-g8qdn\" (UID: \"3630d73a-3222-4dde-a46c-246b5cf499a7\") " pod="openshift-service-ca/service-ca-568869fdc5-g8qdn"
Sep 11 14:41:03.658423 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.658404    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtfv7\" (UniqueName: \"kubernetes.io/projected/3630d73a-3222-4dde-a46c-246b5cf499a7-kube-api-access-vtfv7\") pod \"service-ca-568869fdc5-g8qdn\" (UID: \"3630d73a-3222-4dde-a46c-246b5cf499a7\") " pod="openshift-service-ca/service-ca-568869fdc5-g8qdn"
Sep 11 14:41:03.658983 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.658966    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3630d73a-3222-4dde-a46c-246b5cf499a7-signing-cabundle\") pod \"service-ca-568869fdc5-g8qdn\" (UID: \"3630d73a-3222-4dde-a46c-246b5cf499a7\") " pod="openshift-service-ca/service-ca-568869fdc5-g8qdn"
Sep 11 14:41:03.660702 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.660652    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3630d73a-3222-4dde-a46c-246b5cf499a7-signing-key\") pod \"service-ca-568869fdc5-g8qdn\" (UID: \"3630d73a-3222-4dde-a46c-246b5cf499a7\") " pod="openshift-service-ca/service-ca-568869fdc5-g8qdn"
Sep 11 14:41:03.667709 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.667687    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtfv7\" (UniqueName: \"kubernetes.io/projected/3630d73a-3222-4dde-a46c-246b5cf499a7-kube-api-access-vtfv7\") pod \"service-ca-568869fdc5-g8qdn\" (UID: \"3630d73a-3222-4dde-a46c-246b5cf499a7\") " pod="openshift-service-ca/service-ca-568869fdc5-g8qdn"
Sep 11 14:41:03.744038 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.744005    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-g8qdn"
Sep 11 14:41:03.961974 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.961894    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:41:03.961974 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:03.961931    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-metrics-certs\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:41:03.962194 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:03.962028    2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 14:41:03.962194 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:03.962072    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-metrics-certs podName:80c64211-15e9-4ac3-8660-cb0ba7ae9b90 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:11.962058342 +0000 UTC m=+51.750322631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-metrics-certs") pod "router-default-6dbb8d47db-qzqwf" (UID: "80c64211-15e9-4ac3-8660-cb0ba7ae9b90") : secret "router-metrics-certs-default" not found
Sep 11 14:41:03.962194 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:03.962084    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle podName:80c64211-15e9-4ac3-8660-cb0ba7ae9b90 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:11.962078646 +0000 UTC m=+51.750342923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle") pod "router-default-6dbb8d47db-qzqwf" (UID: "80c64211-15e9-4ac3-8660-cb0ba7ae9b90") : configmap references non-existent config key: service-ca.crt
Sep 11 14:41:04.003743 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:04.003717    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7mlmr_2695284a-7a7f-42ef-9f9a-e152c61edaf3/node-ca/0.log"
Sep 11 14:41:04.050649 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:04.050608    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-g8qdn"]
Sep 11 14:41:04.054334 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:41:04.054299    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3630d73a_3222_4dde_a46c_246b5cf499a7.slice/crio-99b693897cdd46cb5f8fea9acc14597f3ce86bd4a8e4a8235b41b428fadc7e7a WatchSource:0}: Error finding container 99b693897cdd46cb5f8fea9acc14597f3ce86bd4a8e4a8235b41b428fadc7e7a: Status 404 returned error can't find the container with id 99b693897cdd46cb5f8fea9acc14597f3ce86bd4a8e4a8235b41b428fadc7e7a
Sep 11 14:41:04.404809 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:04.404768    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-72l5p_d5e7d1c8-7b1d-494f-9c76-fea7135dd88c/serve-healthcheck-canary/0.log"
Sep 11 14:41:04.769204 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:04.769119    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:04.769360 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:04.769244    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 14:41:04.769360 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:04.769294    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8a46ee3-bfd6-45df-b4f3-191914102098-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:04.769360 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:04.769304    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-rbac-proxy-cm podName:d8a46ee3-bfd6-45df-b4f3-191914102098 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:08.769288237 +0000 UTC m=+48.557552533 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s5plj" (UID: "d8a46ee3-bfd6-45df-b4f3-191914102098") : configmap "kube-rbac-proxy" not found
Sep 11 14:41:04.769530 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:04.769379    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 14:41:04.769530 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:04.769422    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8a46ee3-bfd6-45df-b4f3-191914102098-insights-runtime-extractor-tls podName:d8a46ee3-bfd6-45df-b4f3-191914102098 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:08.769412288 +0000 UTC m=+48.557676566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8a46ee3-bfd6-45df-b4f3-191914102098-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s5plj" (UID: "d8a46ee3-bfd6-45df-b4f3-191914102098") : secret "insights-runtime-extractor-tls" not found
Sep 11 14:41:04.969452 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:04.969414    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-g8qdn" event={"ID":"3630d73a-3222-4dde-a46c-246b5cf499a7","Type":"ContainerStarted","Data":"4c17bf8192a9af5f177af6ca8a37381fbf570f74441c115cf51a15bf15153f57"}
Sep 11 14:41:04.969627 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:04.969468    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-g8qdn" event={"ID":"3630d73a-3222-4dde-a46c-246b5cf499a7","Type":"ContainerStarted","Data":"99b693897cdd46cb5f8fea9acc14597f3ce86bd4a8e4a8235b41b428fadc7e7a"}
Sep 11 14:41:04.987198 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:04.987152    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-g8qdn" podStartSLOduration=1.987136614 podStartE2EDuration="1.987136614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:04.986563538 +0000 UTC m=+44.774827838" watchObservedRunningTime="2026-09-11 14:41:04.987136614 +0000 UTC m=+44.775400910"
Sep 11 14:41:08.799038 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:08.799003    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:08.799397 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:08.799107    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8a46ee3-bfd6-45df-b4f3-191914102098-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:08.799397 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:08.799118    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 14:41:08.799397 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:08.799183    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-rbac-proxy-cm podName:d8a46ee3-bfd6-45df-b4f3-191914102098 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:16.79916599 +0000 UTC m=+56.587430266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s5plj" (UID: "d8a46ee3-bfd6-45df-b4f3-191914102098") : configmap "kube-rbac-proxy" not found
Sep 11 14:41:08.799397 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:08.799269    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 14:41:08.799397 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:08.799315    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8a46ee3-bfd6-45df-b4f3-191914102098-insights-runtime-extractor-tls podName:d8a46ee3-bfd6-45df-b4f3-191914102098 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:16.799302253 +0000 UTC m=+56.587566548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8a46ee3-bfd6-45df-b4f3-191914102098-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s5plj" (UID: "d8a46ee3-bfd6-45df-b4f3-191914102098") : secret "insights-runtime-extractor-tls" not found
Sep 11 14:41:10.415844 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:10.415796    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-metrics-tls\") pod \"dns-default-x6c4c\" (UID: \"a0d57bd1-aa8e-41b5-b3a2-720c733285f1\") " pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:41:10.418059 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:10.418031    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0d57bd1-aa8e-41b5-b3a2-720c733285f1-metrics-tls\") pod \"dns-default-x6c4c\" (UID: \"a0d57bd1-aa8e-41b5-b3a2-720c733285f1\") " pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:41:10.427984 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:10.427964    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:41:10.699446 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:10.699379    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x6c4c"]
Sep 11 14:41:10.702429 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:41:10.702400    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d57bd1_aa8e_41b5_b3a2_720c733285f1.slice/crio-68ee4db3cc48b27acc3d302d630b585efb306d85dcebccc9155eadbbbb225322 WatchSource:0}: Error finding container 68ee4db3cc48b27acc3d302d630b585efb306d85dcebccc9155eadbbbb225322: Status 404 returned error can't find the container with id 68ee4db3cc48b27acc3d302d630b585efb306d85dcebccc9155eadbbbb225322
Sep 11 14:41:10.985691 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:10.985612    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x6c4c" event={"ID":"a0d57bd1-aa8e-41b5-b3a2-720c733285f1","Type":"ContainerStarted","Data":"68ee4db3cc48b27acc3d302d630b585efb306d85dcebccc9155eadbbbb225322"}
Sep 11 14:41:12.029995 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:12.029959    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:41:12.030389 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:12.030011    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-metrics-certs\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:41:12.030389 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:12.030145    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle podName:80c64211-15e9-4ac3-8660-cb0ba7ae9b90 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:28.030120528 +0000 UTC m=+67.818384828 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle") pod "router-default-6dbb8d47db-qzqwf" (UID: "80c64211-15e9-4ac3-8660-cb0ba7ae9b90") : configmap references non-existent config key: service-ca.crt
Sep 11 14:41:12.032589 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:12.032566    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-metrics-certs\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:41:12.994134 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:12.994098    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x6c4c" event={"ID":"a0d57bd1-aa8e-41b5-b3a2-720c733285f1","Type":"ContainerStarted","Data":"810e5b007fe13933537160f981982672264592496d0b1168366a9604e016c5e3"}
Sep 11 14:41:12.994310 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:12.994158    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x6c4c" event={"ID":"a0d57bd1-aa8e-41b5-b3a2-720c733285f1","Type":"ContainerStarted","Data":"0c932f63327f96f0c2f75334da2fcf707473e5d0a11614a0f5049c76abd92e57"}
Sep 11 14:41:12.994310 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:12.994206    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:41:13.016448 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:13.016394    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-x6c4c" podStartSLOduration=17.225200058 podStartE2EDuration="19.016379654s" totalImagesPullingTime="1.791179596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:13.014681858 +0000 UTC m=+52.802946159" watchObservedRunningTime="2026-09-11 14:41:13.016379654 +0000 UTC m=+52.804643953"
Sep 11 14:41:16.868208 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:16.868161    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8a46ee3-bfd6-45df-b4f3-191914102098-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:16.868717 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:16.868310    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:16.868811 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:16.868793    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8a46ee3-bfd6-45df-b4f3-191914102098-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:16.870421 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:16.870401    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8a46ee3-bfd6-45df-b4f3-191914102098-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s5plj\" (UID: \"d8a46ee3-bfd6-45df-b4f3-191914102098\") " pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:16.876345 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:16.876316    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s5plj"
Sep 11 14:41:17.150154 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:17.150083    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s5plj"]
Sep 11 14:41:17.152949 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:41:17.152917    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a46ee3_bfd6_45df_b4f3_191914102098.slice/crio-cf6c5b19b9272ba864c3bf7a85b99eed777b20d5e550394c7b3759305d76ec8a WatchSource:0}: Error finding container cf6c5b19b9272ba864c3bf7a85b99eed777b20d5e550394c7b3759305d76ec8a: Status 404 returned error can't find the container with id cf6c5b19b9272ba864c3bf7a85b99eed777b20d5e550394c7b3759305d76ec8a
Sep 11 14:41:18.011501 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:18.011474    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s5plj" event={"ID":"d8a46ee3-bfd6-45df-b4f3-191914102098","Type":"ContainerStarted","Data":"b78e2f9845527aaace4ef8de2e78a4ca4cb0bdd440a05f827b65cf7e9de24191"}
Sep 11 14:41:18.011800 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:18.011536    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s5plj" event={"ID":"d8a46ee3-bfd6-45df-b4f3-191914102098","Type":"ContainerStarted","Data":"cb324b5350713570e1f3137946c3a6fd040ad2d479274b7786523468e1cf53e2"}
Sep 11 14:41:18.011800 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:18.011568    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s5plj" event={"ID":"d8a46ee3-bfd6-45df-b4f3-191914102098","Type":"ContainerStarted","Data":"cf6c5b19b9272ba864c3bf7a85b99eed777b20d5e550394c7b3759305d76ec8a"}
Sep 11 14:41:20.019641 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:20.019605    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s5plj" event={"ID":"d8a46ee3-bfd6-45df-b4f3-191914102098","Type":"ContainerStarted","Data":"6c79b09520728b92580b592ba951d6bb629d0f55c305b8a82e54fb1f6d466b02"}
Sep 11 14:41:20.044795 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:20.044748    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-s5plj" podStartSLOduration=17.840694216 podStartE2EDuration="20.04473414s" totalImagesPullingTime="2.204039924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:41:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:20.043554819 +0000 UTC m=+59.831819139" watchObservedRunningTime="2026-09-11 14:41:20.04473414 +0000 UTC m=+59.832998439"
Sep 11 14:41:20.929627 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:20.929600    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg2rh"
Sep 11 14:41:23.005876 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:23.005834    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x6c4c"
Sep 11 14:41:26.335296 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:26.335263    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txk6q\" (UniqueName: \"kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q\") pod \"network-check-target-qvvkt\" (UID: \"f2636cfe-b8a3-4c64-94f5-7026d6603aa0\") " pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:41:26.338116 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:26.338095    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 14:41:26.348203 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:26.348181    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 14:41:26.358658 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:26.358634    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txk6q\" (UniqueName: \"kubernetes.io/projected/f2636cfe-b8a3-4c64-94f5-7026d6603aa0-kube-api-access-txk6q\") pod \"network-check-target-qvvkt\" (UID: \"f2636cfe-b8a3-4c64-94f5-7026d6603aa0\") " pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:41:26.435865 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:26.435840    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret\") pod \"global-pull-secret-syncer-sxv9g\" (UID: \"3296271a-8ecd-483d-824e-3b9896f81e3c\") " pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:41:26.435969 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:26.435880    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs\") pod \"network-metrics-daemon-zsqws\" (UID: \"1cb940cd-c21b-4118-ad56-bb823c730604\") " pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:41:26.438890 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:26.438872    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 14:41:26.439610 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:26.439597    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 14:41:26.448672 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:26.448653    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb940cd-c21b-4118-ad56-bb823c730604-metrics-certs\") pod \"network-metrics-daemon-zsqws\" (UID: \"1cb940cd-c21b-4118-ad56-bb823c730604\") " pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:41:26.448778 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:26.448681    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3296271a-8ecd-483d-824e-3b9896f81e3c-original-pull-secret\") pod \"global-pull-secret-syncer-sxv9g\" (UID: \"3296271a-8ecd-483d-824e-3b9896f81e3c\") " pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:41:26.467757 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:26.467736    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxv9g"
Sep 11 14:41:26.477731 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:26.477706    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jjd7d\""
Sep 11 14:41:26.483431 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:26.483413    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xdw2d\""
Sep 11 14:41:26.485506 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:26.485489    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zsqws"
Sep 11 14:41:26.491367 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:26.491350    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:41:26.748422 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:26.748322    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sxv9g"]
Sep 11 14:41:26.751182 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:41:26.751158    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3296271a_8ecd_483d_824e_3b9896f81e3c.slice/crio-30b8bebd4574fa367ebd3996636922b77fb49df8620672dee64747b2fa0d3add WatchSource:0}: Error finding container 30b8bebd4574fa367ebd3996636922b77fb49df8620672dee64747b2fa0d3add: Status 404 returned error can't find the container with id 30b8bebd4574fa367ebd3996636922b77fb49df8620672dee64747b2fa0d3add
Sep 11 14:41:26.775860 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:26.775837    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zsqws"]
Sep 11 14:41:26.779215 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:41:26.779195    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb940cd_c21b_4118_ad56_bb823c730604.slice/crio-ef40a44f7fffa6bb7a4cbafd3ff98ad9ea4903b58e673f578629b9f23bf8ac58 WatchSource:0}: Error finding container ef40a44f7fffa6bb7a4cbafd3ff98ad9ea4903b58e673f578629b9f23bf8ac58: Status 404 returned error can't find the container with id ef40a44f7fffa6bb7a4cbafd3ff98ad9ea4903b58e673f578629b9f23bf8ac58
Sep 11 14:41:26.784443 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:26.784424    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qvvkt"]
Sep 11 14:41:26.786827 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:41:26.786790    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2636cfe_b8a3_4c64_94f5_7026d6603aa0.slice/crio-98c783c0fb9e6e55cca096a8edbcb2aa069929f36584c5f902662520cd234dff WatchSource:0}: Error finding container 98c783c0fb9e6e55cca096a8edbcb2aa069929f36584c5f902662520cd234dff: Status 404 returned error can't find the container with id 98c783c0fb9e6e55cca096a8edbcb2aa069929f36584c5f902662520cd234dff
Sep 11 14:41:27.043714 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.043673    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zsqws" event={"ID":"1cb940cd-c21b-4118-ad56-bb823c730604","Type":"ContainerStarted","Data":"ef40a44f7fffa6bb7a4cbafd3ff98ad9ea4903b58e673f578629b9f23bf8ac58"}
Sep 11 14:41:27.044742 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.044717    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qvvkt" event={"ID":"f2636cfe-b8a3-4c64-94f5-7026d6603aa0","Type":"ContainerStarted","Data":"98c783c0fb9e6e55cca096a8edbcb2aa069929f36584c5f902662520cd234dff"}
Sep 11 14:41:27.045671 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.045651    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sxv9g" event={"ID":"3296271a-8ecd-483d-824e-3b9896f81e3c","Type":"ContainerStarted","Data":"30b8bebd4574fa367ebd3996636922b77fb49df8620672dee64747b2fa0d3add"}
Sep 11 14:41:27.631518 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.631482    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b8b86c954-hwzjb"]
Sep 11 14:41:27.635168 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.635142    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b8b86c954-hwzjb"
Sep 11 14:41:27.639734 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.639316    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 14:41:27.641671 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.640562    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 14:41:27.641671 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.640802    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-9cj2k\""
Sep 11 14:41:27.641671 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.641101    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 14:41:27.641671 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.641241    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 14:41:27.647205 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.646433    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b8b86c954-hwzjb"]
Sep 11 14:41:27.732398 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.732366    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-jmgrx"]
Sep 11 14:41:27.737436 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.737140    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-2md4g"]
Sep 11 14:41:27.737436 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.737367    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jmgrx"
Sep 11 14:41:27.741461 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.741327    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-x62wh\""
Sep 11 14:41:27.741593 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.741520    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 11 14:41:27.741593 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.741540    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 11 14:41:27.741893 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.741875    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-685db9b59b-dbkjb"]
Sep 11 14:41:27.742360 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.742124    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-2md4g"
Sep 11 14:41:27.744402 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.744379    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 14:41:27.745675 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.744869    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.745675 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.745169    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 14:41:27.745675 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.745266    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-72fvw\""
Sep 11 14:41:27.749470 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.747872    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4dms\" (UniqueName: \"kubernetes.io/projected/5ef65c77-c48a-4f91-bebf-8aba5a6ae034-kube-api-access-s4dms\") pod \"managed-serviceaccount-addon-agent-b8b86c954-hwzjb\" (UID: \"5ef65c77-c48a-4f91-bebf-8aba5a6ae034\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b8b86c954-hwzjb"
Sep 11 14:41:27.749470 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.748037    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ef65c77-c48a-4f91-bebf-8aba5a6ae034-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-b8b86c954-hwzjb\" (UID: \"5ef65c77-c48a-4f91-bebf-8aba5a6ae034\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b8b86c954-hwzjb"
Sep 11 14:41:27.749470 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.748472    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 14:41:27.749470 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.748478    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 14:41:27.749470 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.748838    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6xrxr\""
Sep 11 14:41:27.749470 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.748915    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 14:41:27.754969 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.754945    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 14:41:27.772058 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.772000    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-2md4g"]
Sep 11 14:41:27.772182 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.772085    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-685db9b59b-dbkjb"]
Sep 11 14:41:27.773384 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.773366    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-jmgrx"]
Sep 11 14:41:27.848712 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.848679    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-registry-tls\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.848934 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.848724    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-installation-pull-secrets\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.848934 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.848760    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-ca-trust-extracted\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.848934 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.848793    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec7abae9-a91a-4672-8841-c6423f247d94-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-jmgrx\" (UID: \"ec7abae9-a91a-4672-8841-c6423f247d94\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jmgrx"
Sep 11 14:41:27.848934 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.848834    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgf5q\" (UniqueName: \"kubernetes.io/projected/c50428a3-8505-4dff-a5f7-dfc19257b065-kube-api-access-tgf5q\") pod \"downloads-76f99584f4-2md4g\" (UID: \"c50428a3-8505-4dff-a5f7-dfc19257b065\") " pod="openshift-console/downloads-76f99584f4-2md4g"
Sep 11 14:41:27.848934 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.848890    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec7abae9-a91a-4672-8841-c6423f247d94-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jmgrx\" (UID: \"ec7abae9-a91a-4672-8841-c6423f247d94\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jmgrx"
Sep 11 14:41:27.849223 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.848996    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4dms\" (UniqueName: \"kubernetes.io/projected/5ef65c77-c48a-4f91-bebf-8aba5a6ae034-kube-api-access-s4dms\") pod \"managed-serviceaccount-addon-agent-b8b86c954-hwzjb\" (UID: \"5ef65c77-c48a-4f91-bebf-8aba5a6ae034\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b8b86c954-hwzjb"
Sep 11 14:41:27.849223 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.849033    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ef65c77-c48a-4f91-bebf-8aba5a6ae034-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-b8b86c954-hwzjb\" (UID: \"5ef65c77-c48a-4f91-bebf-8aba5a6ae034\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b8b86c954-hwzjb"
Sep 11 14:41:27.849223 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.849071    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bft7m\" (UniqueName: \"kubernetes.io/projected/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-kube-api-access-bft7m\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.849223 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.849109    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-trusted-ca\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.849223 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.849161    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-image-registry-private-configuration\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.849223 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.849191    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-bound-sa-token\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.849501 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.849224    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-registry-certificates\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.852025 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.851981    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ef65c77-c48a-4f91-bebf-8aba5a6ae034-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-b8b86c954-hwzjb\" (UID: \"5ef65c77-c48a-4f91-bebf-8aba5a6ae034\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b8b86c954-hwzjb"
Sep 11 14:41:27.866147 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.866098    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4dms\" (UniqueName: \"kubernetes.io/projected/5ef65c77-c48a-4f91-bebf-8aba5a6ae034-kube-api-access-s4dms\") pod \"managed-serviceaccount-addon-agent-b8b86c954-hwzjb\" (UID: \"5ef65c77-c48a-4f91-bebf-8aba5a6ae034\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b8b86c954-hwzjb"
Sep 11 14:41:27.950047 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.949924    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bft7m\" (UniqueName: \"kubernetes.io/projected/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-kube-api-access-bft7m\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.950047 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.949977    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-trusted-ca\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.950047 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.950027    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-image-registry-private-configuration\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.950434 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.950050    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-bound-sa-token\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.950434 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.950079    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-registry-certificates\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.950434 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.950105    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-registry-tls\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.950434 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.950127    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-installation-pull-secrets\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.950434 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.950159    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-ca-trust-extracted\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.950434 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.950195    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec7abae9-a91a-4672-8841-c6423f247d94-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-jmgrx\" (UID: \"ec7abae9-a91a-4672-8841-c6423f247d94\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jmgrx"
Sep 11 14:41:27.950434 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.950219    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgf5q\" (UniqueName: \"kubernetes.io/projected/c50428a3-8505-4dff-a5f7-dfc19257b065-kube-api-access-tgf5q\") pod \"downloads-76f99584f4-2md4g\" (UID: \"c50428a3-8505-4dff-a5f7-dfc19257b065\") " pod="openshift-console/downloads-76f99584f4-2md4g"
Sep 11 14:41:27.950434 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.950248    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec7abae9-a91a-4672-8841-c6423f247d94-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jmgrx\" (UID: \"ec7abae9-a91a-4672-8841-c6423f247d94\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jmgrx"
Sep 11 14:41:27.951556 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.951187    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-ca-trust-extracted\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.951556 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.951348    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-registry-certificates\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.951965 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.951918    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec7abae9-a91a-4672-8841-c6423f247d94-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-jmgrx\" (UID: \"ec7abae9-a91a-4672-8841-c6423f247d94\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jmgrx"
Sep 11 14:41:27.952168 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.952123    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-trusted-ca\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.953743 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.953688    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ec7abae9-a91a-4672-8841-c6423f247d94-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jmgrx\" (UID: \"ec7abae9-a91a-4672-8841-c6423f247d94\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jmgrx"
Sep 11 14:41:27.956012 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.955970    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-installation-pull-secrets\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.956110 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.956061    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-registry-tls\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.956689 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.956637    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-image-registry-private-configuration\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.959620 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.959432    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b8b86c954-hwzjb"
Sep 11 14:41:27.965739 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.965708    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-bound-sa-token\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.967962 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.967918    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bft7m\" (UniqueName: \"kubernetes.io/projected/9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae-kube-api-access-bft7m\") pod \"image-registry-685db9b59b-dbkjb\" (UID: \"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae\") " pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:27.969544 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:27.969503    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgf5q\" (UniqueName: \"kubernetes.io/projected/c50428a3-8505-4dff-a5f7-dfc19257b065-kube-api-access-tgf5q\") pod \"downloads-76f99584f4-2md4g\" (UID: \"c50428a3-8505-4dff-a5f7-dfc19257b065\") " pod="openshift-console/downloads-76f99584f4-2md4g"
Sep 11 14:41:28.050876 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:28.050841    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:41:28.051563 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:28.051533    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c64211-15e9-4ac3-8660-cb0ba7ae9b90-service-ca-bundle\") pod \"router-default-6dbb8d47db-qzqwf\" (UID: \"80c64211-15e9-4ac3-8660-cb0ba7ae9b90\") " pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:41:28.054795 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:28.054769    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jmgrx"
Sep 11 14:41:28.063960 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:28.063939    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-2md4g"
Sep 11 14:41:28.071144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:28.071123    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:28.309926 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:28.309692    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4dmnm\""
Sep 11 14:41:28.313716 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:28.313692    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:41:28.450428 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:28.450394    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-jmgrx"]
Sep 11 14:41:28.462711 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:28.462683    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b8b86c954-hwzjb"]
Sep 11 14:41:28.466993 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:28.466932    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-685db9b59b-dbkjb"]
Sep 11 14:41:28.468089 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:28.468036    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-2md4g"]
Sep 11 14:41:28.509599 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:41:28.509560    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7abae9_a91a_4672_8841_c6423f247d94.slice/crio-9d1f2d1e4fa9274e2f05183c42b2651883f58df3aebac758fdc6fc9aa3ff5acc WatchSource:0}: Error finding container 9d1f2d1e4fa9274e2f05183c42b2651883f58df3aebac758fdc6fc9aa3ff5acc: Status 404 returned error can't find the container with id 9d1f2d1e4fa9274e2f05183c42b2651883f58df3aebac758fdc6fc9aa3ff5acc
Sep 11 14:41:28.511304 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:41:28.511114    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ef65c77_c48a_4f91_bebf_8aba5a6ae034.slice/crio-1ea789e046fb127e49d5e5e2f39494979deac74cf7a04a170625987e10202316 WatchSource:0}: Error finding container 1ea789e046fb127e49d5e5e2f39494979deac74cf7a04a170625987e10202316: Status 404 returned error can't find the container with id 1ea789e046fb127e49d5e5e2f39494979deac74cf7a04a170625987e10202316
Sep 11 14:41:28.511975 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:41:28.511900    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1f55c3_70e8_4c87_a4f5_7cfb74ba05ae.slice/crio-c63cdac7a678a1676fd94c5c3909219d9f8dac43b95c93b61479b22c9c3c4f56 WatchSource:0}: Error finding container c63cdac7a678a1676fd94c5c3909219d9f8dac43b95c93b61479b22c9c3c4f56: Status 404 returned error can't find the container with id c63cdac7a678a1676fd94c5c3909219d9f8dac43b95c93b61479b22c9c3c4f56
Sep 11 14:41:28.512429 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:41:28.512091    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50428a3_8505_4dff_a5f7_dfc19257b065.slice/crio-4103290294c584362ff4048caf996d7d3bcd36333c4e11e2ec2c52fef6f430d6 WatchSource:0}: Error finding container 4103290294c584362ff4048caf996d7d3bcd36333c4e11e2ec2c52fef6f430d6: Status 404 returned error can't find the container with id 4103290294c584362ff4048caf996d7d3bcd36333c4e11e2ec2c52fef6f430d6
Sep 11 14:41:28.824302 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:28.824209    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6dbb8d47db-qzqwf"]
Sep 11 14:41:28.827365 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:41:28.827335    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c64211_15e9_4ac3_8660_cb0ba7ae9b90.slice/crio-6d79df7c463ce15fc230176981506d1258e624278b781832101f7000d19e4812 WatchSource:0}: Error finding container 6d79df7c463ce15fc230176981506d1258e624278b781832101f7000d19e4812: Status 404 returned error can't find the container with id 6d79df7c463ce15fc230176981506d1258e624278b781832101f7000d19e4812
Sep 11 14:41:29.057743 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:29.057688    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-2md4g" event={"ID":"c50428a3-8505-4dff-a5f7-dfc19257b065","Type":"ContainerStarted","Data":"4103290294c584362ff4048caf996d7d3bcd36333c4e11e2ec2c52fef6f430d6"}
Sep 11 14:41:29.060686 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:29.060119    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zsqws" event={"ID":"1cb940cd-c21b-4118-ad56-bb823c730604","Type":"ContainerStarted","Data":"83308167bc42b77b75f24b72dde48e4f8134279d399cb0c3a42e83b46a11d2a1"}
Sep 11 14:41:29.060686 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:29.060161    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zsqws" event={"ID":"1cb940cd-c21b-4118-ad56-bb823c730604","Type":"ContainerStarted","Data":"3057d748570eab562381495b54c8452ad9097e97e84c49a5dc759b2847f3552b"}
Sep 11 14:41:29.063150 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:29.063070    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-685db9b59b-dbkjb" event={"ID":"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae","Type":"ContainerStarted","Data":"9327d957988249f48331862b4dc7c71e179a76001e52074fdaac4785299074dc"}
Sep 11 14:41:29.063150 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:29.063115    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-685db9b59b-dbkjb" event={"ID":"9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae","Type":"ContainerStarted","Data":"c63cdac7a678a1676fd94c5c3909219d9f8dac43b95c93b61479b22c9c3c4f56"}
Sep 11 14:41:29.063578 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:29.063557    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:29.065143 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:29.065118    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jmgrx" event={"ID":"ec7abae9-a91a-4672-8841-c6423f247d94","Type":"ContainerStarted","Data":"9d1f2d1e4fa9274e2f05183c42b2651883f58df3aebac758fdc6fc9aa3ff5acc"}
Sep 11 14:41:29.067450 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:29.067040    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6dbb8d47db-qzqwf" event={"ID":"80c64211-15e9-4ac3-8660-cb0ba7ae9b90","Type":"ContainerStarted","Data":"a2de828fe16d6d37730f62ef5a8d5f2261b498e7b82ce8ab274034918b048df0"}
Sep 11 14:41:29.067450 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:29.067089    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6dbb8d47db-qzqwf" event={"ID":"80c64211-15e9-4ac3-8660-cb0ba7ae9b90","Type":"ContainerStarted","Data":"6d79df7c463ce15fc230176981506d1258e624278b781832101f7000d19e4812"}
Sep 11 14:41:29.069069 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:29.069028    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b8b86c954-hwzjb" event={"ID":"5ef65c77-c48a-4f91-bebf-8aba5a6ae034","Type":"ContainerStarted","Data":"1ea789e046fb127e49d5e5e2f39494979deac74cf7a04a170625987e10202316"}
Sep 11 14:41:29.108843 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:29.107979    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-685db9b59b-dbkjb" podStartSLOduration=2.107961122 podStartE2EDuration="2.107961122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:29.107297405 +0000 UTC m=+68.895561706" watchObservedRunningTime="2026-09-11 14:41:29.107961122 +0000 UTC m=+68.896225417"
Sep 11 14:41:29.108843 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:29.108125    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zsqws" podStartSLOduration=67.347393178 podStartE2EDuration="1m9.108116149s" totalImagesPullingTime="1.760722971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:29.079166812 +0000 UTC m=+68.867431141" watchObservedRunningTime="2026-09-11 14:41:29.108116149 +0000 UTC m=+68.896380450"
Sep 11 14:41:29.130489 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:29.129023    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6dbb8d47db-qzqwf" podStartSLOduration=33.129008041 podStartE2EDuration="33.129008041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:29.128156104 +0000 UTC m=+68.916420404" watchObservedRunningTime="2026-09-11 14:41:29.129008041 +0000 UTC m=+68.917272340"
Sep 11 14:41:29.314800 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:29.314681    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:41:29.317911 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:29.317691    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:41:30.073914 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:30.073881    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:41:30.075224 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:30.075200    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6dbb8d47db-qzqwf"
Sep 11 14:41:33.089943 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:33.089408    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jmgrx" event={"ID":"ec7abae9-a91a-4672-8841-c6423f247d94","Type":"ContainerStarted","Data":"14cf40e609f845e874ee0c27d27b8e475e55824a4fb2d65a431ed7a9d6774627"}
Sep 11 14:41:33.092004 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:33.091856    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qvvkt" event={"ID":"f2636cfe-b8a3-4c64-94f5-7026d6603aa0","Type":"ContainerStarted","Data":"7429537e8796cd5e279a9097e6e4dbc0682b9f88d76c883059e2162c7db2a44a"}
Sep 11 14:41:33.092004 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:33.091958    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:41:33.093939 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:33.093892    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sxv9g" event={"ID":"3296271a-8ecd-483d-824e-3b9896f81e3c","Type":"ContainerStarted","Data":"5ee6fd2d1fadca340910eea3810c7f27c1b0d87ec52a7bde3298dafb409a68e1"}
Sep 11 14:41:33.108808 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:33.108761    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jmgrx" podStartSLOduration=1.947385375 podStartE2EDuration="6.108744802s" totalImagesPullingTime="4.161359427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:33.107889144 +0000 UTC m=+72.896153465" watchObservedRunningTime="2026-09-11 14:41:33.108744802 +0000 UTC m=+72.897009100"
Sep 11 14:41:33.128957 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:33.128371    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sxv9g" podStartSLOduration=66.242605331 podStartE2EDuration="1m12.12835357s" totalImagesPullingTime="5.885748239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:33.128135729 +0000 UTC m=+72.916400028" watchObservedRunningTime="2026-09-11 14:41:33.12835357 +0000 UTC m=+72.916617869"
Sep 11 14:41:33.146991 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:33.146932    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qvvkt" podStartSLOduration=67.296866803 podStartE2EDuration="1m13.14691276s" totalImagesPullingTime="5.850045957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:33.145685409 +0000 UTC m=+72.933949709" watchObservedRunningTime="2026-09-11 14:41:33.14691276 +0000 UTC m=+72.935177063"
Sep 11 14:41:35.104779 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:35.104735    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b8b86c954-hwzjb" event={"ID":"5ef65c77-c48a-4f91-bebf-8aba5a6ae034","Type":"ContainerStarted","Data":"a17925a798db473c22ed6cc452528794fbb0404385923846dcc7f6a419a0da0d"}
Sep 11 14:41:38.007366 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.007306    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b8b86c954-hwzjb" podStartSLOduration=5.187429234 podStartE2EDuration="11.007282137s" totalImagesPullingTime="5.819852903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:35.12680561 +0000 UTC m=+74.915069911" watchObservedRunningTime="2026-09-11 14:41:38.007282137 +0000 UTC m=+77.795546439"
Sep 11 14:41:38.008384 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.008361    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j"]
Sep 11 14:41:38.011936 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.011912    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j"
Sep 11 14:41:38.015956 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.015934    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 14:41:38.016077 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.015988    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 14:41:38.016077 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.016035    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 14:41:38.016195 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.015934    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 14:41:38.016195 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.015989    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 14:41:38.016195 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.015934    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dtjsf\""
Sep 11 14:41:38.020765 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.020741    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j"]
Sep 11 14:41:38.134918 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.134888    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05f6b2c3-dbe8-48a0-b786-be0e55edcb2f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-vnc5j\" (UID: \"05f6b2c3-dbe8-48a0-b786-be0e55edcb2f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j"
Sep 11 14:41:38.135104 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.134932    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/05f6b2c3-dbe8-48a0-b786-be0e55edcb2f-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-vnc5j\" (UID: \"05f6b2c3-dbe8-48a0-b786-be0e55edcb2f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j"
Sep 11 14:41:38.135104 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.135011    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05f6b2c3-dbe8-48a0-b786-be0e55edcb2f-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-vnc5j\" (UID: \"05f6b2c3-dbe8-48a0-b786-be0e55edcb2f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j"
Sep 11 14:41:38.135104 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.135037    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmhkw\" (UniqueName: \"kubernetes.io/projected/05f6b2c3-dbe8-48a0-b786-be0e55edcb2f-kube-api-access-lmhkw\") pod \"prometheus-operator-68bcd7f987-vnc5j\" (UID: \"05f6b2c3-dbe8-48a0-b786-be0e55edcb2f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j"
Sep 11 14:41:38.236066 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.236032    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05f6b2c3-dbe8-48a0-b786-be0e55edcb2f-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-vnc5j\" (UID: \"05f6b2c3-dbe8-48a0-b786-be0e55edcb2f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j"
Sep 11 14:41:38.236066 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.236069    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmhkw\" (UniqueName: \"kubernetes.io/projected/05f6b2c3-dbe8-48a0-b786-be0e55edcb2f-kube-api-access-lmhkw\") pod \"prometheus-operator-68bcd7f987-vnc5j\" (UID: \"05f6b2c3-dbe8-48a0-b786-be0e55edcb2f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j"
Sep 11 14:41:38.236310 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.236119    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05f6b2c3-dbe8-48a0-b786-be0e55edcb2f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-vnc5j\" (UID: \"05f6b2c3-dbe8-48a0-b786-be0e55edcb2f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j"
Sep 11 14:41:38.236310 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.236169    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/05f6b2c3-dbe8-48a0-b786-be0e55edcb2f-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-vnc5j\" (UID: \"05f6b2c3-dbe8-48a0-b786-be0e55edcb2f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j"
Sep 11 14:41:38.236957 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.236932    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05f6b2c3-dbe8-48a0-b786-be0e55edcb2f-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-vnc5j\" (UID: \"05f6b2c3-dbe8-48a0-b786-be0e55edcb2f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j"
Sep 11 14:41:38.238855 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.238803    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05f6b2c3-dbe8-48a0-b786-be0e55edcb2f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-vnc5j\" (UID: \"05f6b2c3-dbe8-48a0-b786-be0e55edcb2f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j"
Sep 11 14:41:38.238967 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.238854    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/05f6b2c3-dbe8-48a0-b786-be0e55edcb2f-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-vnc5j\" (UID: \"05f6b2c3-dbe8-48a0-b786-be0e55edcb2f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j"
Sep 11 14:41:38.248833 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.248796    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmhkw\" (UniqueName: \"kubernetes.io/projected/05f6b2c3-dbe8-48a0-b786-be0e55edcb2f-kube-api-access-lmhkw\") pod \"prometheus-operator-68bcd7f987-vnc5j\" (UID: \"05f6b2c3-dbe8-48a0-b786-be0e55edcb2f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j"
Sep 11 14:41:38.323069 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.323042    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j"
Sep 11 14:41:38.603863 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:38.603763    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j"]
Sep 11 14:41:38.607196 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:41:38.607166    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f6b2c3_dbe8_48a0_b786_be0e55edcb2f.slice/crio-510d71e6fc4b09b8f2f3495e3b675e76fe61f15257aa633422103dbe8bd08e5e WatchSource:0}: Error finding container 510d71e6fc4b09b8f2f3495e3b675e76fe61f15257aa633422103dbe8bd08e5e: Status 404 returned error can't find the container with id 510d71e6fc4b09b8f2f3495e3b675e76fe61f15257aa633422103dbe8bd08e5e
Sep 11 14:41:39.122044 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:39.122012    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j" event={"ID":"05f6b2c3-dbe8-48a0-b786-be0e55edcb2f","Type":"ContainerStarted","Data":"510d71e6fc4b09b8f2f3495e3b675e76fe61f15257aa633422103dbe8bd08e5e"}
Sep 11 14:41:46.152894 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:46.152782    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-2md4g" event={"ID":"c50428a3-8505-4dff-a5f7-dfc19257b065","Type":"ContainerStarted","Data":"c0bd6572ef5b0d55f872210ca77b83ffaa0c317e79f7984243468ef4bbb36153"}
Sep 11 14:41:47.159244 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:47.159196    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j" event={"ID":"05f6b2c3-dbe8-48a0-b786-be0e55edcb2f","Type":"ContainerStarted","Data":"0ae622793e04a8865f72f51332a8f1e2f4e3fe49c329fefc98c8e4e8a68b152c"}
Sep 11 14:41:47.159739 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:47.159278    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j" event={"ID":"05f6b2c3-dbe8-48a0-b786-be0e55edcb2f","Type":"ContainerStarted","Data":"2eed5f99c672859e6d4c5e4c1090f0f58e5650968f8496ebb44209ebc8f456a8"}
Sep 11 14:41:47.159739 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:47.159335    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-2md4g"
Sep 11 14:41:47.176207 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:47.176176    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-2md4g"
Sep 11 14:41:47.181736 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:47.181681    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-vnc5j" podStartSLOduration=2.072822843 podStartE2EDuration="10.181663806s" totalImagesPullingTime="8.108840963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:47.179910002 +0000 UTC m=+86.968174304" watchObservedRunningTime="2026-09-11 14:41:47.181663806 +0000 UTC m=+86.969928104"
Sep 11 14:41:47.182212 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:47.182166    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-2md4g" podStartSLOduration=3.049452296 podStartE2EDuration="20.182151852s" totalImagesPullingTime="17.132699556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:46.175748261 +0000 UTC m=+85.964012556" watchObservedRunningTime="2026-09-11 14:41:47.182151852 +0000 UTC m=+86.970416152"
Sep 11 14:41:49.430516 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.430472    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7bk7k"]
Sep 11 14:41:49.478835 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.478795    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.482257 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.482232    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 14:41:49.482696 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.482679    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 14:41:49.483561 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.483326    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9pcp2\""
Sep 11 14:41:49.483763 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.483373    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 14:41:49.634732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.634695    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2508fcba-ec05-41e2-a3ca-6490dc704fc3-root\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.635022 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.635000    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2508fcba-ec05-41e2-a3ca-6490dc704fc3-node-exporter-textfile\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.635187 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.635170    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2508fcba-ec05-41e2-a3ca-6490dc704fc3-metrics-client-ca\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.635322 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.635304    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2508fcba-ec05-41e2-a3ca-6490dc704fc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.635451 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.635436    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2508fcba-ec05-41e2-a3ca-6490dc704fc3-sys\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.635574 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.635559    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2508fcba-ec05-41e2-a3ca-6490dc704fc3-node-exporter-wtmp\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.635696 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.635679    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2508fcba-ec05-41e2-a3ca-6490dc704fc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.635808 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.635793    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dp57\" (UniqueName: \"kubernetes.io/projected/2508fcba-ec05-41e2-a3ca-6490dc704fc3-kube-api-access-8dp57\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.635946 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.635931    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2508fcba-ec05-41e2-a3ca-6490dc704fc3-node-exporter-tls\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.736457 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.736367    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2508fcba-ec05-41e2-a3ca-6490dc704fc3-node-exporter-tls\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.736457 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.736438    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2508fcba-ec05-41e2-a3ca-6490dc704fc3-root\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.736679 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.736498    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2508fcba-ec05-41e2-a3ca-6490dc704fc3-node-exporter-textfile\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.736679 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.736562    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2508fcba-ec05-41e2-a3ca-6490dc704fc3-metrics-client-ca\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.736679 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.736559    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2508fcba-ec05-41e2-a3ca-6490dc704fc3-root\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.736679 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.736590    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2508fcba-ec05-41e2-a3ca-6490dc704fc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.736679 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.736634    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2508fcba-ec05-41e2-a3ca-6490dc704fc3-sys\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.736679 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.736665    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2508fcba-ec05-41e2-a3ca-6490dc704fc3-node-exporter-wtmp\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.737006 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.736703    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2508fcba-ec05-41e2-a3ca-6490dc704fc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.737006 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.736731    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dp57\" (UniqueName: \"kubernetes.io/projected/2508fcba-ec05-41e2-a3ca-6490dc704fc3-kube-api-access-8dp57\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.737006 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.736911    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2508fcba-ec05-41e2-a3ca-6490dc704fc3-node-exporter-textfile\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.737006 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.736975    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2508fcba-ec05-41e2-a3ca-6490dc704fc3-sys\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.737350 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.737320    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2508fcba-ec05-41e2-a3ca-6490dc704fc3-node-exporter-wtmp\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.737483 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.737438    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2508fcba-ec05-41e2-a3ca-6490dc704fc3-metrics-client-ca\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.737901 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.737869    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2508fcba-ec05-41e2-a3ca-6490dc704fc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.737991 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:49.737963    2568 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 14:41:49.738100 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:41:49.738031    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2508fcba-ec05-41e2-a3ca-6490dc704fc3-node-exporter-tls podName:2508fcba-ec05-41e2-a3ca-6490dc704fc3 nodeName:}" failed. No retries permitted until 2026-09-11 14:41:50.23801312 +0000 UTC m=+90.026277412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2508fcba-ec05-41e2-a3ca-6490dc704fc3-node-exporter-tls") pod "node-exporter-7bk7k" (UID: "2508fcba-ec05-41e2-a3ca-6490dc704fc3") : secret "node-exporter-tls" not found
Sep 11 14:41:49.742562 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.742538    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2508fcba-ec05-41e2-a3ca-6490dc704fc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:49.764562 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:49.764529    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dp57\" (UniqueName: \"kubernetes.io/projected/2508fcba-ec05-41e2-a3ca-6490dc704fc3-kube-api-access-8dp57\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:50.079856 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.079639    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-685db9b59b-dbkjb"
Sep 11 14:41:50.242093 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.242051    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2508fcba-ec05-41e2-a3ca-6490dc704fc3-node-exporter-tls\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:50.244828 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.244785    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2508fcba-ec05-41e2-a3ca-6490dc704fc3-node-exporter-tls\") pod \"node-exporter-7bk7k\" (UID: \"2508fcba-ec05-41e2-a3ca-6490dc704fc3\") " pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:50.393471 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.393392    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7bk7k"
Sep 11 14:41:50.430090 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:41:50.430057    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2508fcba_ec05_41e2_a3ca_6490dc704fc3.slice/crio-a9a93ef640be438b831062426955403776ec2046c0e6ecea1a20cc1ea088e757 WatchSource:0}: Error finding container a9a93ef640be438b831062426955403776ec2046c0e6ecea1a20cc1ea088e757: Status 404 returned error can't find the container with id a9a93ef640be438b831062426955403776ec2046c0e6ecea1a20cc1ea088e757
Sep 11 14:41:50.611480 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.611449    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 14:41:50.644020 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.643947    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.646875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.646847    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 14:41:50.647181 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.647160    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 14:41:50.647403 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.647384    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-st628\""
Sep 11 14:41:50.647844 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.647809    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 14:41:50.648094 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.648076    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 14:41:50.648643 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.648619    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 14:41:50.648853 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.648835    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 14:41:50.649041 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.648910    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 14:41:50.649157 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.648953    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 14:41:50.656586 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.656565    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 14:41:50.661538 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.661516    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 14:41:50.746458 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.746418    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.746634 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.746462    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-web-config\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.746634 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.746486    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2xfn\" (UniqueName: \"kubernetes.io/projected/cc0805ea-2141-4f44-a021-d4521221b162-kube-api-access-p2xfn\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.746634 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.746579    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.746634 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.746617    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.746806 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.746647    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc0805ea-2141-4f44-a021-d4521221b162-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.746806 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.746719    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc0805ea-2141-4f44-a021-d4521221b162-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.746806 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.746768    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-config-volume\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.746806 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.746790    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.747005 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.746834    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.747005 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.746863    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc0805ea-2141-4f44-a021-d4521221b162-config-out\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.747005 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.746913    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc0805ea-2141-4f44-a021-d4521221b162-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.747005 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.746953    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cc0805ea-2141-4f44-a021-d4521221b162-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.847875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.847839    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.848037 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.847891    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc0805ea-2141-4f44-a021-d4521221b162-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.848037 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.847959    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc0805ea-2141-4f44-a021-d4521221b162-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.848037 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.848006    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-config-volume\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.848037 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.848028    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.848396 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.848366    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.848480 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.848420    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc0805ea-2141-4f44-a021-d4521221b162-config-out\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.848480 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.848454    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc0805ea-2141-4f44-a021-d4521221b162-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.848588 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.848485    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cc0805ea-2141-4f44-a021-d4521221b162-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.848588 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.848531    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.848588 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.848553    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-web-config\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.848588 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.848575    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2xfn\" (UniqueName: \"kubernetes.io/projected/cc0805ea-2141-4f44-a021-d4521221b162-kube-api-access-p2xfn\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.848783 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.848642    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.848862 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.848809    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc0805ea-2141-4f44-a021-d4521221b162-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.851212 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.851081    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.851524 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.851499    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.854224 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.854010    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.854977 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.854383    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-web-config\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.855542 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.855519    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc0805ea-2141-4f44-a021-d4521221b162-config-out\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.855793 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.855752    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cc0805ea-2141-4f44-a021-d4521221b162-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.855913 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.855876    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc0805ea-2141-4f44-a021-d4521221b162-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.856547 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.856286    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.856547 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.856391    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-config-volume\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.856678 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.856604    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc0805ea-2141-4f44-a021-d4521221b162-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.857319 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.857297    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cc0805ea-2141-4f44-a021-d4521221b162-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.862019 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.861991    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2xfn\" (UniqueName: \"kubernetes.io/projected/cc0805ea-2141-4f44-a021-d4521221b162-kube-api-access-p2xfn\") pod \"alertmanager-main-0\" (UID: \"cc0805ea-2141-4f44-a021-d4521221b162\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:50.958834 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:50.958739    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 14:41:51.181347 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.181304    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7bk7k" event={"ID":"2508fcba-ec05-41e2-a3ca-6490dc704fc3","Type":"ContainerStarted","Data":"a9a93ef640be438b831062426955403776ec2046c0e6ecea1a20cc1ea088e757"}
Sep 11 14:41:51.280594 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.280550    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 14:41:51.462921 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.462885    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7f8767cffd-drhbp"]
Sep 11 14:41:51.492383 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.492312    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f8767cffd-drhbp"]
Sep 11 14:41:51.492592 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.492456    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.496087 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.496062    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mhjnd\""
Sep 11 14:41:51.496201 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.496120    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 11 14:41:51.496268 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.496048    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 11 14:41:51.496529 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.496326    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 11 14:41:51.496529 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.496415    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 11 14:41:51.496529 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.496518    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3b29d5h9lj8gn\""
Sep 11 14:41:51.496763 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.496641    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 11 14:41:51.591642 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:41:51.591561    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0805ea_2141_4f44_a021_d4521221b162.slice/crio-75b5469a54264be0d1455d596b959cfc3319c1f859c3ee3fb83301a315face8c WatchSource:0}: Error finding container 75b5469a54264be0d1455d596b959cfc3319c1f859c3ee3fb83301a315face8c: Status 404 returned error can't find the container with id 75b5469a54264be0d1455d596b959cfc3319c1f859c3ee3fb83301a315face8c
Sep 11 14:41:51.656101 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.656074    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/97afa455-97c4-47f8-9045-15a06c0d6d10-secret-grpc-tls\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.656490 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.656121    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtjsx\" (UniqueName: \"kubernetes.io/projected/97afa455-97c4-47f8-9045-15a06c0d6d10-kube-api-access-qtjsx\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.656490 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.656186    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/97afa455-97c4-47f8-9045-15a06c0d6d10-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.656490 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.656255    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97afa455-97c4-47f8-9045-15a06c0d6d10-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.656490 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.656290    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/97afa455-97c4-47f8-9045-15a06c0d6d10-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.656490 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.656327    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/97afa455-97c4-47f8-9045-15a06c0d6d10-secret-thanos-querier-tls\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.656490 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.656354    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97afa455-97c4-47f8-9045-15a06c0d6d10-metrics-client-ca\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.656490 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.656423    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97afa455-97c4-47f8-9045-15a06c0d6d10-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.757799 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.757756    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97afa455-97c4-47f8-9045-15a06c0d6d10-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.758006 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.757838    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/97afa455-97c4-47f8-9045-15a06c0d6d10-secret-grpc-tls\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.758006 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.757880    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtjsx\" (UniqueName: \"kubernetes.io/projected/97afa455-97c4-47f8-9045-15a06c0d6d10-kube-api-access-qtjsx\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.758006 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.757909    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/97afa455-97c4-47f8-9045-15a06c0d6d10-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.758006 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.757947    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97afa455-97c4-47f8-9045-15a06c0d6d10-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.758006 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.757972    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/97afa455-97c4-47f8-9045-15a06c0d6d10-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.758263 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.758006    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/97afa455-97c4-47f8-9045-15a06c0d6d10-secret-thanos-querier-tls\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.758263 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.758033    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97afa455-97c4-47f8-9045-15a06c0d6d10-metrics-client-ca\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.758780 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.758755    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97afa455-97c4-47f8-9045-15a06c0d6d10-metrics-client-ca\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.760890 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.760852    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97afa455-97c4-47f8-9045-15a06c0d6d10-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.761138 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.761067    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/97afa455-97c4-47f8-9045-15a06c0d6d10-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.761138 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.761126    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/97afa455-97c4-47f8-9045-15a06c0d6d10-secret-grpc-tls\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.761464 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.761431    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97afa455-97c4-47f8-9045-15a06c0d6d10-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.761767 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.761737    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/97afa455-97c4-47f8-9045-15a06c0d6d10-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.761908 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.761856    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/97afa455-97c4-47f8-9045-15a06c0d6d10-secret-thanos-querier-tls\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.768304 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.768281    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtjsx\" (UniqueName: \"kubernetes.io/projected/97afa455-97c4-47f8-9045-15a06c0d6d10-kube-api-access-qtjsx\") pod \"thanos-querier-7f8767cffd-drhbp\" (UID: \"97afa455-97c4-47f8-9045-15a06c0d6d10\") " pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:51.806138 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:51.806104    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:52.187718 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:52.187671    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7bk7k" event={"ID":"2508fcba-ec05-41e2-a3ca-6490dc704fc3","Type":"ContainerStarted","Data":"324279ece5e26b8a3852b836909c3d3b1c63479fffb3e6bc099b8093734a3ea8"}
Sep 11 14:41:52.189169 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:52.189140    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc0805ea-2141-4f44-a021-d4521221b162","Type":"ContainerStarted","Data":"75b5469a54264be0d1455d596b959cfc3319c1f859c3ee3fb83301a315face8c"}
Sep 11 14:41:52.245714 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:52.245678    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f8767cffd-drhbp"]
Sep 11 14:41:52.248324 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:41:52.248293    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97afa455_97c4_47f8_9045_15a06c0d6d10.slice/crio-9901df6845a59b95e6d3709a0c2f7251eb9670b87301de8ec05ed2a3fd8fbbfc WatchSource:0}: Error finding container 9901df6845a59b95e6d3709a0c2f7251eb9670b87301de8ec05ed2a3fd8fbbfc: Status 404 returned error can't find the container with id 9901df6845a59b95e6d3709a0c2f7251eb9670b87301de8ec05ed2a3fd8fbbfc
Sep 11 14:41:53.195535 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:53.195493    2568 generic.go:356] "Generic (PLEG): container finished" podID="2508fcba-ec05-41e2-a3ca-6490dc704fc3" containerID="324279ece5e26b8a3852b836909c3d3b1c63479fffb3e6bc099b8093734a3ea8" exitCode=0
Sep 11 14:41:53.196280 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:53.195653    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7bk7k" event={"ID":"2508fcba-ec05-41e2-a3ca-6490dc704fc3","Type":"ContainerDied","Data":"324279ece5e26b8a3852b836909c3d3b1c63479fffb3e6bc099b8093734a3ea8"}
Sep 11 14:41:53.198519 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:53.197341    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp" event={"ID":"97afa455-97c4-47f8-9045-15a06c0d6d10","Type":"ContainerStarted","Data":"9901df6845a59b95e6d3709a0c2f7251eb9670b87301de8ec05ed2a3fd8fbbfc"}
Sep 11 14:41:54.173525 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:54.172124    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-b2m7b"]
Sep 11 14:41:54.197660 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:54.197624    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-b2m7b"]
Sep 11 14:41:54.198155 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:54.197680    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b2m7b"
Sep 11 14:41:54.200780 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:54.200753    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dp5mn\""
Sep 11 14:41:54.200916 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:54.200900    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 11 14:41:54.219235 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:54.219207    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7bk7k" event={"ID":"2508fcba-ec05-41e2-a3ca-6490dc704fc3","Type":"ContainerStarted","Data":"fd6ff2bb026f8d23d46ed59995c61779ce4ec45db9262842b6fd1aa03c5183e8"}
Sep 11 14:41:54.219353 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:54.219271    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7bk7k" event={"ID":"2508fcba-ec05-41e2-a3ca-6490dc704fc3","Type":"ContainerStarted","Data":"de2d86d02f89ea78dbc6d5829500ecb0d9c23127ca1c74354943dbcf664e5ac2"}
Sep 11 14:41:54.222684 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:54.222142    2568 generic.go:356] "Generic (PLEG): container finished" podID="cc0805ea-2141-4f44-a021-d4521221b162" containerID="5fafe9bb9152460f8c3ded927719653619378e2a620c1aa863d6f0d8392bca87" exitCode=0
Sep 11 14:41:54.222684 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:54.222243    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc0805ea-2141-4f44-a021-d4521221b162","Type":"ContainerDied","Data":"5fafe9bb9152460f8c3ded927719653619378e2a620c1aa863d6f0d8392bca87"}
Sep 11 14:41:54.243054 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:54.242965    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7bk7k" podStartSLOduration=3.629495971 podStartE2EDuration="5.242946169s" totalImagesPullingTime="1.531487775s" totalInitContainerRuntime="81.962423ms" isStatefulPod=true podCreationTimestamp="2026-09-11 14:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:54.241629778 +0000 UTC m=+94.029894077" watchObservedRunningTime="2026-09-11 14:41:54.242946169 +0000 UTC m=+94.031210475"
Sep 11 14:41:54.282508 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:54.282475    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzvtt\" (UniqueName: \"kubernetes.io/projected/7f331a12-a765-450a-97ac-626f2f44aada-kube-api-access-xzvtt\") pod \"monitoring-plugin-5d85778b9d-b2m7b\" (UID: \"7f331a12-a765-450a-97ac-626f2f44aada\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b2m7b"
Sep 11 14:41:54.282715 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:54.282533    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f331a12-a765-450a-97ac-626f2f44aada-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-b2m7b\" (UID: \"7f331a12-a765-450a-97ac-626f2f44aada\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b2m7b"
Sep 11 14:41:54.384233 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:54.384161    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f331a12-a765-450a-97ac-626f2f44aada-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-b2m7b\" (UID: \"7f331a12-a765-450a-97ac-626f2f44aada\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b2m7b"
Sep 11 14:41:54.384608 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:54.384504    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzvtt\" (UniqueName: \"kubernetes.io/projected/7f331a12-a765-450a-97ac-626f2f44aada-kube-api-access-xzvtt\") pod \"monitoring-plugin-5d85778b9d-b2m7b\" (UID: \"7f331a12-a765-450a-97ac-626f2f44aada\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b2m7b"
Sep 11 14:41:54.387329 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:54.387304    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f331a12-a765-450a-97ac-626f2f44aada-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-b2m7b\" (UID: \"7f331a12-a765-450a-97ac-626f2f44aada\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b2m7b"
Sep 11 14:41:54.402637 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:54.402580    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzvtt\" (UniqueName: \"kubernetes.io/projected/7f331a12-a765-450a-97ac-626f2f44aada-kube-api-access-xzvtt\") pod \"monitoring-plugin-5d85778b9d-b2m7b\" (UID: \"7f331a12-a765-450a-97ac-626f2f44aada\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b2m7b"
Sep 11 14:41:54.519017 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:54.518946    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b2m7b"
Sep 11 14:41:54.850790 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:54.850767    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-b2m7b"]
Sep 11 14:41:54.968154 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:41:54.968114    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f331a12_a765_450a_97ac_626f2f44aada.slice/crio-f1c1a3a055c9bb3bf6954e151b305428c1e1efe892782b79e05c6e86e9ffe6c8 WatchSource:0}: Error finding container f1c1a3a055c9bb3bf6954e151b305428c1e1efe892782b79e05c6e86e9ffe6c8: Status 404 returned error can't find the container with id f1c1a3a055c9bb3bf6954e151b305428c1e1efe892782b79e05c6e86e9ffe6c8
Sep 11 14:41:55.228853 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:55.228796    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp" event={"ID":"97afa455-97c4-47f8-9045-15a06c0d6d10","Type":"ContainerStarted","Data":"d2fb70b997b7ed7e45a2df97534d880ee3d5a89775c469802e253429a9b053ee"}
Sep 11 14:41:55.230295 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:55.230270    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b2m7b" event={"ID":"7f331a12-a765-450a-97ac-626f2f44aada","Type":"ContainerStarted","Data":"f1c1a3a055c9bb3bf6954e151b305428c1e1efe892782b79e05c6e86e9ffe6c8"}
Sep 11 14:41:56.236405 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:56.236369    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp" event={"ID":"97afa455-97c4-47f8-9045-15a06c0d6d10","Type":"ContainerStarted","Data":"ba205eaab327c55cfe9b4dec2a01cd8ace075cc20968a86a7d8f5b4176d92497"}
Sep 11 14:41:56.236744 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:56.236435    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp" event={"ID":"97afa455-97c4-47f8-9045-15a06c0d6d10","Type":"ContainerStarted","Data":"923783e6a9886479205c84322544d899ed9b09f139d2e57b3fbc71b63004f9df"}
Sep 11 14:41:57.244829 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:57.244761    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc0805ea-2141-4f44-a021-d4521221b162","Type":"ContainerStarted","Data":"ab62c37c362aef66e57c1b1084011359fb301327d8b7941ecb0cc1d7ba054595"}
Sep 11 14:41:57.245291 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:57.244914    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc0805ea-2141-4f44-a021-d4521221b162","Type":"ContainerStarted","Data":"43a5880865791c088752ddd62606cb8c22f893efe81faef8cf9d077a174446ba"}
Sep 11 14:41:57.245291 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:57.244963    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc0805ea-2141-4f44-a021-d4521221b162","Type":"ContainerStarted","Data":"66a6da4b07824fa1674cddc7e436d6399afeac2dda3f86343c83e73f2e75534a"}
Sep 11 14:41:58.253684 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:58.253595    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b2m7b" event={"ID":"7f331a12-a765-450a-97ac-626f2f44aada","Type":"ContainerStarted","Data":"42dd63e3bdf087d5261acd197971250ef313170b57baf79038e7965377d10363"}
Sep 11 14:41:58.253684 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:58.253663    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b2m7b"
Sep 11 14:41:58.257077 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:58.257046    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp" event={"ID":"97afa455-97c4-47f8-9045-15a06c0d6d10","Type":"ContainerStarted","Data":"1f196020675cd2d64021d0efcbc3b024ba4c2e76c74cbc3916ba6dc80b0131e4"}
Sep 11 14:41:58.257207 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:58.257123    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp" event={"ID":"97afa455-97c4-47f8-9045-15a06c0d6d10","Type":"ContainerStarted","Data":"5546773be24edc4d7293cc69027e01775f053e6c949dea50753ed4bda4c7175f"}
Sep 11 14:41:58.257207 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:58.257175    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:41:58.257310 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:58.257215    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp" event={"ID":"97afa455-97c4-47f8-9045-15a06c0d6d10","Type":"ContainerStarted","Data":"68c64da6e173618a1bdb4966e19b33292628d4fb18157cabe7f554a75b6afa55"}
Sep 11 14:41:58.258846 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:58.258805    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b2m7b"
Sep 11 14:41:58.260909 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:58.260889    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc0805ea-2141-4f44-a021-d4521221b162","Type":"ContainerStarted","Data":"d28c0d5560ac4d1b85275197c68508fb684cf4e557a93d920a5a1fd7a99d53e2"}
Sep 11 14:41:58.261020 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:58.260954    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc0805ea-2141-4f44-a021-d4521221b162","Type":"ContainerStarted","Data":"ca7a20135128a9b1cf864c731e6fd719ad6c3db54010ed9ee2160333c0eec088"}
Sep 11 14:41:58.261020 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:58.260999    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc0805ea-2141-4f44-a021-d4521221b162","Type":"ContainerStarted","Data":"2a1ba0e6322465f1c292291a38cbbd22f07970fe37bf2614b618976725e0e5bd"}
Sep 11 14:41:58.273693 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:58.273651    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b2m7b" podStartSLOduration=1.434993338 podStartE2EDuration="4.273637693s" totalImagesPullingTime="2.838644355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:58.272376751 +0000 UTC m=+98.060641075" watchObservedRunningTime="2026-09-11 14:41:58.273637693 +0000 UTC m=+98.061901991"
Sep 11 14:41:58.338285 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:58.338233    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.267075692 podStartE2EDuration="8.338193136s" totalImagesPullingTime="4.04491293s" totalInitContainerRuntime="26.204514ms" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:58.337052095 +0000 UTC m=+98.125316394" watchObservedRunningTime="2026-09-11 14:41:58.338193136 +0000 UTC m=+98.126457434"
Sep 11 14:41:58.338938 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:41:58.338907    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp" podStartSLOduration=2.46118999 podStartE2EDuration="7.338898327s" totalImagesPullingTime="4.877708337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:58.302865836 +0000 UTC m=+98.091130135" watchObservedRunningTime="2026-09-11 14:41:58.338898327 +0000 UTC m=+98.127162672"
Sep 11 14:42:04.102162 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:42:04.102128    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qvvkt"
Sep 11 14:42:04.271711 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:42:04.271681    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7f8767cffd-drhbp"
Sep 11 14:42:30.385218 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:42:30.385186    2568 generic.go:356] "Generic (PLEG): container finished" podID="a12f3fa6-b08b-436f-bef3-ebe50265cc52" containerID="5f2919bb55b98631148b0a8d588f5b3fbf1c3c05f08b9a8bd5778a2bc723b2cd" exitCode=0
Sep 11 14:42:30.385657 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:42:30.385269    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rbgf9" event={"ID":"a12f3fa6-b08b-436f-bef3-ebe50265cc52","Type":"ContainerDied","Data":"5f2919bb55b98631148b0a8d588f5b3fbf1c3c05f08b9a8bd5778a2bc723b2cd"}
Sep 11 14:42:30.385724 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:42:30.385667    2568 scope.go:117] "RemoveContainer" containerID="5f2919bb55b98631148b0a8d588f5b3fbf1c3c05f08b9a8bd5778a2bc723b2cd"
Sep 11 14:42:31.390625 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:42:31.390584    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rbgf9" event={"ID":"a12f3fa6-b08b-436f-bef3-ebe50265cc52","Type":"ContainerStarted","Data":"da3cbe972b268377c0cabffcae5b27375a106bd1d7bcddb3099fe47d7f415b91"}
Sep 11 14:44:45.930086 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:44:45.929998    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dfxpw"]
Sep 11 14:44:45.933190 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:44:45.933175    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-dfxpw"
Sep 11 14:44:45.937160 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:44:45.937127    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-whr7x\""
Sep 11 14:44:45.938048 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:44:45.938030    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 14:44:45.938107 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:44:45.938050    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 14:44:45.961266 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:44:45.961243    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dfxpw"]
Sep 11 14:44:46.004494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:44:46.004457    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crppb\" (UniqueName: \"kubernetes.io/projected/93e4b2d2-0d4f-41d7-99d2-658d439474fe-kube-api-access-crppb\") pod \"cert-manager-webhook-6486ffd997-dfxpw\" (UID: \"93e4b2d2-0d4f-41d7-99d2-658d439474fe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dfxpw"
Sep 11 14:44:46.004628 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:44:46.004569    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93e4b2d2-0d4f-41d7-99d2-658d439474fe-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dfxpw\" (UID: \"93e4b2d2-0d4f-41d7-99d2-658d439474fe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dfxpw"
Sep 11 14:44:46.105889 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:44:46.105855    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crppb\" (UniqueName: \"kubernetes.io/projected/93e4b2d2-0d4f-41d7-99d2-658d439474fe-kube-api-access-crppb\") pod \"cert-manager-webhook-6486ffd997-dfxpw\" (UID: \"93e4b2d2-0d4f-41d7-99d2-658d439474fe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dfxpw"
Sep 11 14:44:46.106032 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:44:46.105990    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93e4b2d2-0d4f-41d7-99d2-658d439474fe-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dfxpw\" (UID: \"93e4b2d2-0d4f-41d7-99d2-658d439474fe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dfxpw"
Sep 11 14:44:46.120701 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:44:46.120676    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93e4b2d2-0d4f-41d7-99d2-658d439474fe-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dfxpw\" (UID: \"93e4b2d2-0d4f-41d7-99d2-658d439474fe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dfxpw"
Sep 11 14:44:46.122150 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:44:46.122129    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crppb\" (UniqueName: \"kubernetes.io/projected/93e4b2d2-0d4f-41d7-99d2-658d439474fe-kube-api-access-crppb\") pod \"cert-manager-webhook-6486ffd997-dfxpw\" (UID: \"93e4b2d2-0d4f-41d7-99d2-658d439474fe\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dfxpw"
Sep 11 14:44:46.241589 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:44:46.241520    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-dfxpw"
Sep 11 14:44:46.527727 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:44:46.527700    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dfxpw"]
Sep 11 14:44:46.529585 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:44:46.529554    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93e4b2d2_0d4f_41d7_99d2_658d439474fe.slice/crio-dee199d341ccbb0f23d718c627addbae3d4bed20a1c3dc2d7a08a62a38938c22 WatchSource:0}: Error finding container dee199d341ccbb0f23d718c627addbae3d4bed20a1c3dc2d7a08a62a38938c22: Status 404 returned error can't find the container with id dee199d341ccbb0f23d718c627addbae3d4bed20a1c3dc2d7a08a62a38938c22
Sep 11 14:44:46.911604 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:44:46.911524    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-dfxpw" event={"ID":"93e4b2d2-0d4f-41d7-99d2-658d439474fe","Type":"ContainerStarted","Data":"dee199d341ccbb0f23d718c627addbae3d4bed20a1c3dc2d7a08a62a38938c22"}
Sep 11 14:44:49.927140 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:44:49.927110    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-dfxpw" event={"ID":"93e4b2d2-0d4f-41d7-99d2-658d439474fe","Type":"ContainerStarted","Data":"e428d6b72d4c99396b4ab98f5af4fe09db9d606f5a65ff32d79d7ffab7c3c51f"}
Sep 11 14:44:49.927512 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:44:49.927229    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-dfxpw"
Sep 11 14:44:49.946209 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:44:49.946159    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-dfxpw" podStartSLOduration=1.624264709 podStartE2EDuration="4.946138096s" totalImagesPullingTime="3.321873387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:44:49.943942767 +0000 UTC m=+269.732207066" watchObservedRunningTime="2026-09-11 14:44:49.946138096 +0000 UTC m=+269.734402399"
Sep 11 14:44:55.933844 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:44:55.933784    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-dfxpw"
Sep 11 14:45:05.871537 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:05.871503    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2vcvd"]
Sep 11 14:45:05.878043 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:05.878020    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-2vcvd"
Sep 11 14:45:05.878541 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:05.878518    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2vcvd"]
Sep 11 14:45:05.880732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:05.880710    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gx9qt\""
Sep 11 14:45:05.964833 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:05.964789    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05fb26b7-562a-47dc-93e6-96c7127a6cd6-bound-sa-token\") pod \"cert-manager-55b4d954d5-2vcvd\" (UID: \"05fb26b7-562a-47dc-93e6-96c7127a6cd6\") " pod="cert-manager/cert-manager-55b4d954d5-2vcvd"
Sep 11 14:45:05.964987 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:05.964859    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbghg\" (UniqueName: \"kubernetes.io/projected/05fb26b7-562a-47dc-93e6-96c7127a6cd6-kube-api-access-qbghg\") pod \"cert-manager-55b4d954d5-2vcvd\" (UID: \"05fb26b7-562a-47dc-93e6-96c7127a6cd6\") " pod="cert-manager/cert-manager-55b4d954d5-2vcvd"
Sep 11 14:45:06.065866 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:06.065808    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05fb26b7-562a-47dc-93e6-96c7127a6cd6-bound-sa-token\") pod \"cert-manager-55b4d954d5-2vcvd\" (UID: \"05fb26b7-562a-47dc-93e6-96c7127a6cd6\") " pod="cert-manager/cert-manager-55b4d954d5-2vcvd"
Sep 11 14:45:06.066012 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:06.065876    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbghg\" (UniqueName: \"kubernetes.io/projected/05fb26b7-562a-47dc-93e6-96c7127a6cd6-kube-api-access-qbghg\") pod \"cert-manager-55b4d954d5-2vcvd\" (UID: \"05fb26b7-562a-47dc-93e6-96c7127a6cd6\") " pod="cert-manager/cert-manager-55b4d954d5-2vcvd"
Sep 11 14:45:06.077711 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:06.077681    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05fb26b7-562a-47dc-93e6-96c7127a6cd6-bound-sa-token\") pod \"cert-manager-55b4d954d5-2vcvd\" (UID: \"05fb26b7-562a-47dc-93e6-96c7127a6cd6\") " pod="cert-manager/cert-manager-55b4d954d5-2vcvd"
Sep 11 14:45:06.077844 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:06.077752    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbghg\" (UniqueName: \"kubernetes.io/projected/05fb26b7-562a-47dc-93e6-96c7127a6cd6-kube-api-access-qbghg\") pod \"cert-manager-55b4d954d5-2vcvd\" (UID: \"05fb26b7-562a-47dc-93e6-96c7127a6cd6\") " pod="cert-manager/cert-manager-55b4d954d5-2vcvd"
Sep 11 14:45:06.187497 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:06.187416    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-2vcvd"
Sep 11 14:45:06.461811 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:06.461751    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2vcvd"]
Sep 11 14:45:06.464404 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:45:06.464369    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05fb26b7_562a_47dc_93e6_96c7127a6cd6.slice/crio-3b977f3035679cca6a4b287b4449f94b5afdc13ffc8f473d112022b695237c65 WatchSource:0}: Error finding container 3b977f3035679cca6a4b287b4449f94b5afdc13ffc8f473d112022b695237c65: Status 404 returned error can't find the container with id 3b977f3035679cca6a4b287b4449f94b5afdc13ffc8f473d112022b695237c65
Sep 11 14:45:06.993253 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:06.993218    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-2vcvd" event={"ID":"05fb26b7-562a-47dc-93e6-96c7127a6cd6","Type":"ContainerStarted","Data":"509e61c9a3320ac2ca924f5244770bbc06485de57a9fd8fec2fcf8d8683e9742"}
Sep 11 14:45:06.993625 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:06.993265    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-2vcvd" event={"ID":"05fb26b7-562a-47dc-93e6-96c7127a6cd6","Type":"ContainerStarted","Data":"3b977f3035679cca6a4b287b4449f94b5afdc13ffc8f473d112022b695237c65"}
Sep 11 14:45:07.010869 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:07.010804    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-2vcvd" podStartSLOduration=2.010789201 podStartE2EDuration="2.010789201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:07.009963907 +0000 UTC m=+286.798228205" watchObservedRunningTime="2026-09-11 14:45:07.010789201 +0000 UTC m=+286.799053499"
Sep 11 14:45:20.555636 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:20.555610    2568 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 14:45:23.951370 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:23.951332    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"]
Sep 11 14:45:23.954619 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:23.954599    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"
Sep 11 14:45:23.958083 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:23.958056    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 14:45:23.958211 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:23.958191    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 14:45:23.958261 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:23.958233    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zjcsv\""
Sep 11 14:45:23.958261 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:23.958247    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 14:45:23.958528 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:23.958510    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 14:45:23.974765 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:23.974742    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"]
Sep 11 14:45:24.106584 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:24.106548    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/25614aed-8da5-40cc-a40b-e9a13d6a623a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f987f856-gcmcx\" (UID: \"25614aed-8da5-40cc-a40b-e9a13d6a623a\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"
Sep 11 14:45:24.106761 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:24.106596    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/25614aed-8da5-40cc-a40b-e9a13d6a623a-manifests\") pod \"opendatahub-operator-controller-manager-67f987f856-gcmcx\" (UID: \"25614aed-8da5-40cc-a40b-e9a13d6a623a\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"
Sep 11 14:45:24.106761 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:24.106662    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25614aed-8da5-40cc-a40b-e9a13d6a623a-tmp\") pod \"opendatahub-operator-controller-manager-67f987f856-gcmcx\" (UID: \"25614aed-8da5-40cc-a40b-e9a13d6a623a\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"
Sep 11 14:45:24.106761 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:24.106716    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25614aed-8da5-40cc-a40b-e9a13d6a623a-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f987f856-gcmcx\" (UID: \"25614aed-8da5-40cc-a40b-e9a13d6a623a\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"
Sep 11 14:45:24.106761 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:24.106731    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8wx4\" (UniqueName: \"kubernetes.io/projected/25614aed-8da5-40cc-a40b-e9a13d6a623a-kube-api-access-t8wx4\") pod \"opendatahub-operator-controller-manager-67f987f856-gcmcx\" (UID: \"25614aed-8da5-40cc-a40b-e9a13d6a623a\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"
Sep 11 14:45:24.207415 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:24.207315    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25614aed-8da5-40cc-a40b-e9a13d6a623a-tmp\") pod \"opendatahub-operator-controller-manager-67f987f856-gcmcx\" (UID: \"25614aed-8da5-40cc-a40b-e9a13d6a623a\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"
Sep 11 14:45:24.207578 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:24.207457    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25614aed-8da5-40cc-a40b-e9a13d6a623a-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f987f856-gcmcx\" (UID: \"25614aed-8da5-40cc-a40b-e9a13d6a623a\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"
Sep 11 14:45:24.207578 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:24.207480    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8wx4\" (UniqueName: \"kubernetes.io/projected/25614aed-8da5-40cc-a40b-e9a13d6a623a-kube-api-access-t8wx4\") pod \"opendatahub-operator-controller-manager-67f987f856-gcmcx\" (UID: \"25614aed-8da5-40cc-a40b-e9a13d6a623a\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"
Sep 11 14:45:24.207578 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:24.207502    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/25614aed-8da5-40cc-a40b-e9a13d6a623a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f987f856-gcmcx\" (UID: \"25614aed-8da5-40cc-a40b-e9a13d6a623a\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"
Sep 11 14:45:24.207578 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:24.207525    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/25614aed-8da5-40cc-a40b-e9a13d6a623a-manifests\") pod \"opendatahub-operator-controller-manager-67f987f856-gcmcx\" (UID: \"25614aed-8da5-40cc-a40b-e9a13d6a623a\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"
Sep 11 14:45:24.207861 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:24.207844    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/25614aed-8da5-40cc-a40b-e9a13d6a623a-manifests\") pod \"opendatahub-operator-controller-manager-67f987f856-gcmcx\" (UID: \"25614aed-8da5-40cc-a40b-e9a13d6a623a\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"
Sep 11 14:45:24.209498 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:24.209471    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25614aed-8da5-40cc-a40b-e9a13d6a623a-tmp\") pod \"opendatahub-operator-controller-manager-67f987f856-gcmcx\" (UID: \"25614aed-8da5-40cc-a40b-e9a13d6a623a\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"
Sep 11 14:45:24.209964 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:24.209946    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/25614aed-8da5-40cc-a40b-e9a13d6a623a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f987f856-gcmcx\" (UID: \"25614aed-8da5-40cc-a40b-e9a13d6a623a\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"
Sep 11 14:45:24.210018 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:24.209976    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25614aed-8da5-40cc-a40b-e9a13d6a623a-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f987f856-gcmcx\" (UID: \"25614aed-8da5-40cc-a40b-e9a13d6a623a\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"
Sep 11 14:45:24.217116 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:24.217096    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8wx4\" (UniqueName: \"kubernetes.io/projected/25614aed-8da5-40cc-a40b-e9a13d6a623a-kube-api-access-t8wx4\") pod \"opendatahub-operator-controller-manager-67f987f856-gcmcx\" (UID: \"25614aed-8da5-40cc-a40b-e9a13d6a623a\") " pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"
Sep 11 14:45:24.264621 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:24.264590    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"
Sep 11 14:45:24.553677 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:24.553647    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"]
Sep 11 14:45:24.557808 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:45:24.557783    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25614aed_8da5_40cc_a40b_e9a13d6a623a.slice/crio-79671e58e6f997fe3156941bca4d784a0a5c09ec0dbb1c86c03ba37739548d7b WatchSource:0}: Error finding container 79671e58e6f997fe3156941bca4d784a0a5c09ec0dbb1c86c03ba37739548d7b: Status 404 returned error can't find the container with id 79671e58e6f997fe3156941bca4d784a0a5c09ec0dbb1c86c03ba37739548d7b
Sep 11 14:45:24.559544 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:24.559526    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:45:25.070869 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:25.070834    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx" event={"ID":"25614aed-8da5-40cc-a40b-e9a13d6a623a","Type":"ContainerStarted","Data":"79671e58e6f997fe3156941bca4d784a0a5c09ec0dbb1c86c03ba37739548d7b"}
Sep 11 14:45:28.085927 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:28.085891    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx" event={"ID":"25614aed-8da5-40cc-a40b-e9a13d6a623a","Type":"ContainerStarted","Data":"e4455a377889f2cdf41cd91aa8c08dbf4fde66ef383700dbb821a9a4a8359dbe"}
Sep 11 14:45:29.091655 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:29.091617    2568 generic.go:356] "Generic (PLEG): container finished" podID="25614aed-8da5-40cc-a40b-e9a13d6a623a" containerID="e4455a377889f2cdf41cd91aa8c08dbf4fde66ef383700dbb821a9a4a8359dbe" exitCode=0
Sep 11 14:45:29.092140 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:29.091683    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx" event={"ID":"25614aed-8da5-40cc-a40b-e9a13d6a623a","Type":"ContainerDied","Data":"e4455a377889f2cdf41cd91aa8c08dbf4fde66ef383700dbb821a9a4a8359dbe"}
Sep 11 14:45:30.097609 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:30.097575    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx" event={"ID":"25614aed-8da5-40cc-a40b-e9a13d6a623a","Type":"ContainerStarted","Data":"0d7efc37786e1df8f3aa53a1a81dd9f7379b00ddcad0a8d25bb3620c0c3a4af9"}
Sep 11 14:45:30.098007 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:30.097713    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"
Sep 11 14:45:41.104475 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:41.104445    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx"
Sep 11 14:45:41.129408 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:41.129358    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67f987f856-gcmcx" podStartSLOduration=13.941566574 podStartE2EDuration="18.129342642s" totalImagesPullingTime="3.163016317s" totalInitContainerRuntime="1.024759751s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:30.129811678 +0000 UTC m=+309.918075976" watchObservedRunningTime="2026-09-11 14:45:41.129342642 +0000 UTC m=+320.917606941"
Sep 11 14:45:43.892618 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:43.892577    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-789999dc4-rh6hh"]
Sep 11 14:45:43.895831 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:43.895804    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-789999dc4-rh6hh"
Sep 11 14:45:43.898541 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:43.898519    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 14:45:43.899659 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:43.899642    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-bp2n9\""
Sep 11 14:45:43.899718 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:43.899672    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 14:45:43.905521 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:43.905500    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-789999dc4-rh6hh"]
Sep 11 14:45:43.952426 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:43.952400    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh49w\" (UniqueName: \"kubernetes.io/projected/cc4ddb02-a957-436b-8345-f0bdabe0d5d0-kube-api-access-sh49w\") pod \"kube-auth-proxy-789999dc4-rh6hh\" (UID: \"cc4ddb02-a957-436b-8345-f0bdabe0d5d0\") " pod="openshift-ingress/kube-auth-proxy-789999dc4-rh6hh"
Sep 11 14:45:43.952536 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:43.952458    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc4ddb02-a957-436b-8345-f0bdabe0d5d0-tmp\") pod \"kube-auth-proxy-789999dc4-rh6hh\" (UID: \"cc4ddb02-a957-436b-8345-f0bdabe0d5d0\") " pod="openshift-ingress/kube-auth-proxy-789999dc4-rh6hh"
Sep 11 14:45:43.952536 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:43.952481    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc4ddb02-a957-436b-8345-f0bdabe0d5d0-tls-certs\") pod \"kube-auth-proxy-789999dc4-rh6hh\" (UID: \"cc4ddb02-a957-436b-8345-f0bdabe0d5d0\") " pod="openshift-ingress/kube-auth-proxy-789999dc4-rh6hh"
Sep 11 14:45:44.053419 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:44.053392    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc4ddb02-a957-436b-8345-f0bdabe0d5d0-tmp\") pod \"kube-auth-proxy-789999dc4-rh6hh\" (UID: \"cc4ddb02-a957-436b-8345-f0bdabe0d5d0\") " pod="openshift-ingress/kube-auth-proxy-789999dc4-rh6hh"
Sep 11 14:45:44.053563 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:44.053426    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc4ddb02-a957-436b-8345-f0bdabe0d5d0-tls-certs\") pod \"kube-auth-proxy-789999dc4-rh6hh\" (UID: \"cc4ddb02-a957-436b-8345-f0bdabe0d5d0\") " pod="openshift-ingress/kube-auth-proxy-789999dc4-rh6hh"
Sep 11 14:45:44.053563 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:44.053469    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh49w\" (UniqueName: \"kubernetes.io/projected/cc4ddb02-a957-436b-8345-f0bdabe0d5d0-kube-api-access-sh49w\") pod \"kube-auth-proxy-789999dc4-rh6hh\" (UID: \"cc4ddb02-a957-436b-8345-f0bdabe0d5d0\") " pod="openshift-ingress/kube-auth-proxy-789999dc4-rh6hh"
Sep 11 14:45:44.055643 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:44.055612    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc4ddb02-a957-436b-8345-f0bdabe0d5d0-tmp\") pod \"kube-auth-proxy-789999dc4-rh6hh\" (UID: \"cc4ddb02-a957-436b-8345-f0bdabe0d5d0\") " pod="openshift-ingress/kube-auth-proxy-789999dc4-rh6hh"
Sep 11 14:45:44.055763 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:44.055712    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc4ddb02-a957-436b-8345-f0bdabe0d5d0-tls-certs\") pod \"kube-auth-proxy-789999dc4-rh6hh\" (UID: \"cc4ddb02-a957-436b-8345-f0bdabe0d5d0\") " pod="openshift-ingress/kube-auth-proxy-789999dc4-rh6hh"
Sep 11 14:45:44.068136 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:44.068113    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh49w\" (UniqueName: \"kubernetes.io/projected/cc4ddb02-a957-436b-8345-f0bdabe0d5d0-kube-api-access-sh49w\") pod \"kube-auth-proxy-789999dc4-rh6hh\" (UID: \"cc4ddb02-a957-436b-8345-f0bdabe0d5d0\") " pod="openshift-ingress/kube-auth-proxy-789999dc4-rh6hh"
Sep 11 14:45:44.205286 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:44.205208    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-789999dc4-rh6hh"
Sep 11 14:45:44.488246 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:44.488219    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-789999dc4-rh6hh"]
Sep 11 14:45:44.490717 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:45:44.490689    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4ddb02_a957_436b_8345_f0bdabe0d5d0.slice/crio-7d397770e8231af46e68ab6f8f124e7754d023d778890394a0c990193c721832 WatchSource:0}: Error finding container 7d397770e8231af46e68ab6f8f124e7754d023d778890394a0c990193c721832: Status 404 returned error can't find the container with id 7d397770e8231af46e68ab6f8f124e7754d023d778890394a0c990193c721832
Sep 11 14:45:45.171958 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:45.171916    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-789999dc4-rh6hh" event={"ID":"cc4ddb02-a957-436b-8345-f0bdabe0d5d0","Type":"ContainerStarted","Data":"7d397770e8231af46e68ab6f8f124e7754d023d778890394a0c990193c721832"}
Sep 11 14:45:45.883221 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:45.883187    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"]
Sep 11 14:45:45.888695 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:45.888659    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:45.892440 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:45.892418    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 14:45:45.892790 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:45.892767    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 14:45:45.892906 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:45.892776    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 14:45:45.892966 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:45.892918    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 14:45:45.893188 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:45.893169    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7bhwz\""
Sep 11 14:45:45.899903 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:45.899880    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"]
Sep 11 14:45:45.970024 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:45.969986    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:45.970212 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:45.970074    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:45.970212 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:45.970104    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:45.970212 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:45.970138    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:45.970384 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:45.970227    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:45.970384 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:45.970266    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:45.970384 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:45.970290    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p98h\" (UniqueName: \"kubernetes.io/projected/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-kube-api-access-7p98h\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:46.071192 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:46.071158    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:46.071374 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:46.071214    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:46.071374 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:46.071241    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p98h\" (UniqueName: \"kubernetes.io/projected/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-kube-api-access-7p98h\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:46.071374 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:46.071278    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:46.071374 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:46.071346    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:46.071374 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:46.071372    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:46.071612 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:46.071417    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:46.072195 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:46.071808    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:46.074356 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:46.074310    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:46.074873 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:46.074768    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:46.074873 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:46.074831    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:46.075008 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:46.074884    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:46.088218 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:46.088193    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p98h\" (UniqueName: \"kubernetes.io/projected/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-kube-api-access-7p98h\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:46.088344 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:46.088280    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-svzvx\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:46.200720 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:46.200637    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:47.428812 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:47.428785    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"]
Sep 11 14:45:47.431225 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:45:47.431191    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70a52fb_fc28_48ee_a413_7893e0c5f6a8.slice/crio-57bd4736e2ff56b73b13fb36b274792f0f4dc6344981d941acb61db0a5abe318 WatchSource:0}: Error finding container 57bd4736e2ff56b73b13fb36b274792f0f4dc6344981d941acb61db0a5abe318: Status 404 returned error can't find the container with id 57bd4736e2ff56b73b13fb36b274792f0f4dc6344981d941acb61db0a5abe318
Sep 11 14:45:48.194158 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:48.194122    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-789999dc4-rh6hh" event={"ID":"cc4ddb02-a957-436b-8345-f0bdabe0d5d0","Type":"ContainerStarted","Data":"1c0ae00775ae537bf91a13c22d956afd451eaf1f3db1cf5116adc893b559dcca"}
Sep 11 14:45:48.195429 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:48.195403    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx" event={"ID":"e70a52fb-fc28-48ee-a413-7893e0c5f6a8","Type":"ContainerStarted","Data":"57bd4736e2ff56b73b13fb36b274792f0f4dc6344981d941acb61db0a5abe318"}
Sep 11 14:45:48.214529 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:48.214473    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-789999dc4-rh6hh" podStartSLOduration=2.497309966 podStartE2EDuration="5.214453829s" totalImagesPullingTime="2.717143863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:48.213301159 +0000 UTC m=+328.001565458" watchObservedRunningTime="2026-09-11 14:45:48.214453829 +0000 UTC m=+328.002718132"
Sep 11 14:45:49.941898 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:49.941856    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:45:49.942203 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:49.941925    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:45:50.206279 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:50.206194    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx" event={"ID":"e70a52fb-fc28-48ee-a413-7893e0c5f6a8","Type":"ContainerStarted","Data":"9761b0632db98d5272c5a6e086b95ef5759ba21344d9a234f667af0255175cb2"}
Sep 11 14:45:50.206534 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:50.206514    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:45:50.208157 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:50.208131    2568 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-svzvx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 14:45:50.208270 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:50.208177    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx" podUID="e70a52fb-fc28-48ee-a413-7893e0c5f6a8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 14:45:50.230768 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:50.230719    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx" podStartSLOduration=2.7223588640000003 podStartE2EDuration="5.230705634s" totalImagesPullingTime="2.50834677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:50.22859739 +0000 UTC m=+330.016861689" watchObservedRunningTime="2026-09-11 14:45:50.230705634 +0000 UTC m=+330.018969932"
Sep 11 14:45:51.213064 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:51.213009    2568 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-svzvx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 14:45:51.213580 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:51.213095    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx" podUID="e70a52fb-fc28-48ee-a413-7893e0c5f6a8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 14:45:54.213531 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:45:54.213498    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:46:02.105493 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.105453    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"]
Sep 11 14:46:02.109028 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.109009    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.132837 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.132789    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"]
Sep 11 14:46:02.204158 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.204124    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8e63873-4c72-46a7-ba0d-0e872730d99b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.204363 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.204173    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffw2z\" (UniqueName: \"kubernetes.io/projected/f8e63873-4c72-46a7-ba0d-0e872730d99b-kube-api-access-ffw2z\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.204363 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.204198    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f8e63873-4c72-46a7-ba0d-0e872730d99b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.204363 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.204301    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8e63873-4c72-46a7-ba0d-0e872730d99b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.204363 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.204338    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f8e63873-4c72-46a7-ba0d-0e872730d99b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.204363 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.204363    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f8e63873-4c72-46a7-ba0d-0e872730d99b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.204562 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.204402    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f8e63873-4c72-46a7-ba0d-0e872730d99b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.305009 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.304968    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8e63873-4c72-46a7-ba0d-0e872730d99b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.305009 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.305004    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f8e63873-4c72-46a7-ba0d-0e872730d99b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.305237 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.305020    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f8e63873-4c72-46a7-ba0d-0e872730d99b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.305237 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.305054    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f8e63873-4c72-46a7-ba0d-0e872730d99b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.305237 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.305081    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8e63873-4c72-46a7-ba0d-0e872730d99b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.305237 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.305113    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffw2z\" (UniqueName: \"kubernetes.io/projected/f8e63873-4c72-46a7-ba0d-0e872730d99b-kube-api-access-ffw2z\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.305237 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.305140    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f8e63873-4c72-46a7-ba0d-0e872730d99b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.305878 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.305845    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f8e63873-4c72-46a7-ba0d-0e872730d99b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.307483 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.307461    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f8e63873-4c72-46a7-ba0d-0e872730d99b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.307586 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.307562    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8e63873-4c72-46a7-ba0d-0e872730d99b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.307634 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.307619    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f8e63873-4c72-46a7-ba0d-0e872730d99b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.307669 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.307640    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f8e63873-4c72-46a7-ba0d-0e872730d99b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.313534 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.313511    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8e63873-4c72-46a7-ba0d-0e872730d99b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.313768 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.313749    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffw2z\" (UniqueName: \"kubernetes.io/projected/f8e63873-4c72-46a7-ba0d-0e872730d99b-kube-api-access-ffw2z\") pod \"istiod-openshift-gateway-5677ccb49b-x8fxp\" (UID: \"f8e63873-4c72-46a7-ba0d-0e872730d99b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.417960 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.417874    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:02.701503 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.701478    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"]
Sep 11 14:46:02.703578 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:46:02.703549    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e63873_4c72_46a7_ba0d_0e872730d99b.slice/crio-04fc5411baa12171fc3321e80f8f4226d6f04605bf3e0abc7bf7d80bb66e8d2a WatchSource:0}: Error finding container 04fc5411baa12171fc3321e80f8f4226d6f04605bf3e0abc7bf7d80bb66e8d2a: Status 404 returned error can't find the container with id 04fc5411baa12171fc3321e80f8f4226d6f04605bf3e0abc7bf7d80bb66e8d2a
Sep 11 14:46:02.705879 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.705843    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:46:02.705976 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:02.705920    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:46:03.265964 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:03.265917    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp" event={"ID":"f8e63873-4c72-46a7-ba0d-0e872730d99b","Type":"ContainerStarted","Data":"7d9ea722f1c6b7a87d4bebd28d672ed6cf14ecee9fc9397d84d5f875967d89f8"}
Sep 11 14:46:03.266438 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:03.265989    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp" event={"ID":"f8e63873-4c72-46a7-ba0d-0e872730d99b","Type":"ContainerStarted","Data":"04fc5411baa12171fc3321e80f8f4226d6f04605bf3e0abc7bf7d80bb66e8d2a"}
Sep 11 14:46:03.266438 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:03.266014    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:03.268162 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:03.268126    2568 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-x8fxp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 14:46:03.268299 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:03.268185    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp" podUID="f8e63873-4c72-46a7-ba0d-0e872730d99b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 14:46:03.293390 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:03.293335    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp" podStartSLOduration=1.293318103 podStartE2EDuration="1.293318103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:46:03.291702571 +0000 UTC m=+343.079966891" watchObservedRunningTime="2026-09-11 14:46:03.293318103 +0000 UTC m=+343.081582401"
Sep 11 14:46:04.270578 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.270548    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-x8fxp"
Sep 11 14:46:04.346657 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.346622    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"]
Sep 11 14:46:04.346970 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.346914    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx" podUID="e70a52fb-fc28-48ee-a413-7893e0c5f6a8" containerName="discovery" containerID="cri-o://9761b0632db98d5272c5a6e086b95ef5759ba21344d9a234f667af0255175cb2" gracePeriod=30
Sep 11 14:46:04.595647 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.595625    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:46:04.727144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.727110    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-local-certs\") pod \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") "
Sep 11 14:46:04.727144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.727149    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-token\") pod \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") "
Sep 11 14:46:04.727349 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.727166    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-csr-dns-cert\") pod \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") "
Sep 11 14:46:04.727349 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.727201    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-cacerts\") pod \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") "
Sep 11 14:46:04.727349 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.727223    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-kubeconfig\") pod \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") "
Sep 11 14:46:04.727349 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.727245    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-csr-ca-configmap\") pod \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") "
Sep 11 14:46:04.727349 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.727342    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7p98h\" (UniqueName: \"kubernetes.io/projected/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-kube-api-access-7p98h\") pod \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\" (UID: \"e70a52fb-fc28-48ee-a413-7893e0c5f6a8\") "
Sep 11 14:46:04.727762 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.727726    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "e70a52fb-fc28-48ee-a413-7893e0c5f6a8" (UID: "e70a52fb-fc28-48ee-a413-7893e0c5f6a8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:46:04.730051 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.730006    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "e70a52fb-fc28-48ee-a413-7893e0c5f6a8" (UID: "e70a52fb-fc28-48ee-a413-7893e0c5f6a8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:46:04.730051 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.730027    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-local-certs" (OuterVolumeSpecName: "local-certs") pod "e70a52fb-fc28-48ee-a413-7893e0c5f6a8" (UID: "e70a52fb-fc28-48ee-a413-7893e0c5f6a8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:46:04.730051 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.730042    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-cacerts" (OuterVolumeSpecName: "cacerts") pod "e70a52fb-fc28-48ee-a413-7893e0c5f6a8" (UID: "e70a52fb-fc28-48ee-a413-7893e0c5f6a8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:46:04.730208 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.730060    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "e70a52fb-fc28-48ee-a413-7893e0c5f6a8" (UID: "e70a52fb-fc28-48ee-a413-7893e0c5f6a8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:46:04.730208 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.730079    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-token" (OuterVolumeSpecName: "istio-token") pod "e70a52fb-fc28-48ee-a413-7893e0c5f6a8" (UID: "e70a52fb-fc28-48ee-a413-7893e0c5f6a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:46:04.730208 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.730194    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-kube-api-access-7p98h" (OuterVolumeSpecName: "kube-api-access-7p98h") pod "e70a52fb-fc28-48ee-a413-7893e0c5f6a8" (UID: "e70a52fb-fc28-48ee-a413-7893e0c5f6a8"). InnerVolumeSpecName "kube-api-access-7p98h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:46:04.828023 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.827952    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p98h\" (UniqueName: \"kubernetes.io/projected/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-kube-api-access-7p98h\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:46:04.828023 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.827978    2568 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-local-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:46:04.828023 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.827992    2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-token\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:46:04.828023 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.828003    2568 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-csr-dns-cert\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:46:04.828023 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.828012    2568 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-cacerts\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:46:04.828023 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.828020    2568 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-kubeconfig\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:46:04.828023 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:04.828029    2568 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e70a52fb-fc28-48ee-a413-7893e0c5f6a8-istio-csr-ca-configmap\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:46:05.276322 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.276137    2568 generic.go:356] "Generic (PLEG): container finished" podID="e70a52fb-fc28-48ee-a413-7893e0c5f6a8" containerID="9761b0632db98d5272c5a6e086b95ef5759ba21344d9a234f667af0255175cb2" exitCode=0
Sep 11 14:46:05.276322 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.276207    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"
Sep 11 14:46:05.276322 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.276292    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx" event={"ID":"e70a52fb-fc28-48ee-a413-7893e0c5f6a8","Type":"ContainerDied","Data":"9761b0632db98d5272c5a6e086b95ef5759ba21344d9a234f667af0255175cb2"}
Sep 11 14:46:05.276915 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.276341    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx" event={"ID":"e70a52fb-fc28-48ee-a413-7893e0c5f6a8","Type":"ContainerDied","Data":"57bd4736e2ff56b73b13fb36b274792f0f4dc6344981d941acb61db0a5abe318"}
Sep 11 14:46:05.276915 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.276386    2568 scope.go:117] "RemoveContainer" containerID="9761b0632db98d5272c5a6e086b95ef5759ba21344d9a234f667af0255175cb2"
Sep 11 14:46:05.286659 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.286638    2568 scope.go:117] "RemoveContainer" containerID="9761b0632db98d5272c5a6e086b95ef5759ba21344d9a234f667af0255175cb2"
Sep 11 14:46:05.287006 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:46:05.286981    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9761b0632db98d5272c5a6e086b95ef5759ba21344d9a234f667af0255175cb2\": container with ID starting with 9761b0632db98d5272c5a6e086b95ef5759ba21344d9a234f667af0255175cb2 not found: ID does not exist" containerID="9761b0632db98d5272c5a6e086b95ef5759ba21344d9a234f667af0255175cb2"
Sep 11 14:46:05.287120 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.287012    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9761b0632db98d5272c5a6e086b95ef5759ba21344d9a234f667af0255175cb2"} err="failed to get container status \"9761b0632db98d5272c5a6e086b95ef5759ba21344d9a234f667af0255175cb2\": rpc error: code = NotFound desc = could not find container \"9761b0632db98d5272c5a6e086b95ef5759ba21344d9a234f667af0255175cb2\": container with ID starting with 9761b0632db98d5272c5a6e086b95ef5759ba21344d9a234f667af0255175cb2 not found: ID does not exist"
Sep 11 14:46:05.308882 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.308808    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"]
Sep 11 14:46:05.315347 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.315320    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-svzvx"]
Sep 11 14:46:05.386195 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.386165    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2trt7"]
Sep 11 14:46:05.386563 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.386545    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e70a52fb-fc28-48ee-a413-7893e0c5f6a8" containerName="discovery"
Sep 11 14:46:05.386853 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.386566    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e70a52fb-fc28-48ee-a413-7893e0c5f6a8" containerName="discovery"
Sep 11 14:46:05.386853 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.386654    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e70a52fb-fc28-48ee-a413-7893e0c5f6a8" containerName="discovery"
Sep 11 14:46:05.391138 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.391116    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2trt7"
Sep 11 14:46:05.394071 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.394054    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-p7w72\""
Sep 11 14:46:05.394188 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.394157    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 11 14:46:05.404750 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.404731    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2trt7"]
Sep 11 14:46:05.433738 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.433707    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77a11e5d-f419-4603-ad16-f076c917205b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2trt7\" (UID: \"77a11e5d-f419-4603-ad16-f076c917205b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2trt7"
Sep 11 14:46:05.433869 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.433835    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjlbz\" (UniqueName: \"kubernetes.io/projected/77a11e5d-f419-4603-ad16-f076c917205b-kube-api-access-tjlbz\") pod \"llmisvc-controller-manager-777c6bd7d5-2trt7\" (UID: \"77a11e5d-f419-4603-ad16-f076c917205b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2trt7"
Sep 11 14:46:05.534968 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.534889    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjlbz\" (UniqueName: \"kubernetes.io/projected/77a11e5d-f419-4603-ad16-f076c917205b-kube-api-access-tjlbz\") pod \"llmisvc-controller-manager-777c6bd7d5-2trt7\" (UID: \"77a11e5d-f419-4603-ad16-f076c917205b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2trt7"
Sep 11 14:46:05.535114 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.534976    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77a11e5d-f419-4603-ad16-f076c917205b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2trt7\" (UID: \"77a11e5d-f419-4603-ad16-f076c917205b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2trt7"
Sep 11 14:46:05.537285 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.537263    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77a11e5d-f419-4603-ad16-f076c917205b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2trt7\" (UID: \"77a11e5d-f419-4603-ad16-f076c917205b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2trt7"
Sep 11 14:46:05.547542 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.547520    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjlbz\" (UniqueName: \"kubernetes.io/projected/77a11e5d-f419-4603-ad16-f076c917205b-kube-api-access-tjlbz\") pod \"llmisvc-controller-manager-777c6bd7d5-2trt7\" (UID: \"77a11e5d-f419-4603-ad16-f076c917205b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2trt7"
Sep 11 14:46:05.700598 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.700563    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2trt7"
Sep 11 14:46:05.974940 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:05.974916    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2trt7"]
Sep 11 14:46:05.977010 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:46:05.976977    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod77a11e5d_f419_4603_ad16_f076c917205b.slice/crio-caf4d5fc3dfa4de04cf485a02896b0667f0447d7c3654a1cb6df349446b8e112 WatchSource:0}: Error finding container caf4d5fc3dfa4de04cf485a02896b0667f0447d7c3654a1cb6df349446b8e112: Status 404 returned error can't find the container with id caf4d5fc3dfa4de04cf485a02896b0667f0447d7c3654a1cb6df349446b8e112
Sep 11 14:46:06.281824 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:06.281789    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2trt7" event={"ID":"77a11e5d-f419-4603-ad16-f076c917205b","Type":"ContainerStarted","Data":"caf4d5fc3dfa4de04cf485a02896b0667f0447d7c3654a1cb6df349446b8e112"}
Sep 11 14:46:06.662593 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:06.662522    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e70a52fb-fc28-48ee-a413-7893e0c5f6a8" path="/var/lib/kubelet/pods/e70a52fb-fc28-48ee-a413-7893e0c5f6a8/volumes"
Sep 11 14:46:09.299483 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:09.299438    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2trt7" event={"ID":"77a11e5d-f419-4603-ad16-f076c917205b","Type":"ContainerStarted","Data":"3005fac3ce1f466d83d8c2d618a1cf664656d91056916a3900656936ee7838e8"}
Sep 11 14:46:09.299884 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:09.299532    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2trt7"
Sep 11 14:46:09.320627 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:09.320582    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2trt7" podStartSLOduration=1.691383556 podStartE2EDuration="4.320567488s" totalImagesPullingTime="2.629183932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:46:09.318772836 +0000 UTC m=+349.107037154" watchObservedRunningTime="2026-09-11 14:46:09.320567488 +0000 UTC m=+349.108831787"
Sep 11 14:46:40.306920 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:46:40.306885    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2trt7"
Sep 11 14:47:11.645409 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:11.645374    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-kh5nc"]
Sep 11 14:47:11.648804 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:11.648787    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-kh5nc"
Sep 11 14:47:11.651429 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:11.651407    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 14:47:11.651429 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:11.651419    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-snwt7\""
Sep 11 14:47:11.651604 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:11.651584    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 14:47:11.660334 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:11.660312    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-kh5nc"]
Sep 11 14:47:11.777144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:11.777108    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftbxz\" (UniqueName: \"kubernetes.io/projected/6f70628e-94b5-44ab-a9f0-78b4bcacdb17-kube-api-access-ftbxz\") pod \"authorino-operator-76b478bd78-kh5nc\" (UID: \"6f70628e-94b5-44ab-a9f0-78b4bcacdb17\") " pod="kuadrant-system/authorino-operator-76b478bd78-kh5nc"
Sep 11 14:47:11.877545 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:11.877509    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftbxz\" (UniqueName: \"kubernetes.io/projected/6f70628e-94b5-44ab-a9f0-78b4bcacdb17-kube-api-access-ftbxz\") pod \"authorino-operator-76b478bd78-kh5nc\" (UID: \"6f70628e-94b5-44ab-a9f0-78b4bcacdb17\") " pod="kuadrant-system/authorino-operator-76b478bd78-kh5nc"
Sep 11 14:47:11.888668 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:11.888637    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftbxz\" (UniqueName: \"kubernetes.io/projected/6f70628e-94b5-44ab-a9f0-78b4bcacdb17-kube-api-access-ftbxz\") pod \"authorino-operator-76b478bd78-kh5nc\" (UID: \"6f70628e-94b5-44ab-a9f0-78b4bcacdb17\") " pod="kuadrant-system/authorino-operator-76b478bd78-kh5nc"
Sep 11 14:47:11.959353 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:11.959269    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-kh5nc"
Sep 11 14:47:12.233263 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:12.233237    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-kh5nc"]
Sep 11 14:47:12.235725 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:47:12.235700    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f70628e_94b5_44ab_a9f0_78b4bcacdb17.slice/crio-aef075ae6c69bc18e1519497f83f2b5b188d336297b5cf4bfb12181bec825062 WatchSource:0}: Error finding container aef075ae6c69bc18e1519497f83f2b5b188d336297b5cf4bfb12181bec825062: Status 404 returned error can't find the container with id aef075ae6c69bc18e1519497f83f2b5b188d336297b5cf4bfb12181bec825062
Sep 11 14:47:12.580413 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:12.580381    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-kh5nc" event={"ID":"6f70628e-94b5-44ab-a9f0-78b4bcacdb17","Type":"ContainerStarted","Data":"aef075ae6c69bc18e1519497f83f2b5b188d336297b5cf4bfb12181bec825062"}
Sep 11 14:47:15.598269 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:15.598230    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-kh5nc" event={"ID":"6f70628e-94b5-44ab-a9f0-78b4bcacdb17","Type":"ContainerStarted","Data":"d3b76afde2ad8c16ad692821dc1e24a2fe794272a303005cf8723b4b67d7c8c2"}
Sep 11 14:47:15.598658 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:15.598316    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-kh5nc"
Sep 11 14:47:15.624551 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:15.624439    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-kh5nc" podStartSLOduration=2.149506108 podStartE2EDuration="4.62442327s" totalImagesPullingTime="2.474917162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:47:15.623850965 +0000 UTC m=+415.412115261" watchObservedRunningTime="2026-09-11 14:47:15.62442327 +0000 UTC m=+415.412687570"
Sep 11 14:47:26.606304 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:26.606272    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-kh5nc"
Sep 11 14:47:37.724461 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:37.724428    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5"]
Sep 11 14:47:37.731240 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:37.731219    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5"
Sep 11 14:47:37.734497 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:37.734473    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qns29\""
Sep 11 14:47:37.749259 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:37.749234    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5"]
Sep 11 14:47:37.791148 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:37.791118    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnwvn\" (UniqueName: \"kubernetes.io/projected/b788786c-b8dd-4a50-99ac-33eb5546445d-kube-api-access-tnwvn\") pod \"kuadrant-operator-controller-manager-5f685c496f-797m5\" (UID: \"b788786c-b8dd-4a50-99ac-33eb5546445d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5"
Sep 11 14:47:37.791301 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:37.791161    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b788786c-b8dd-4a50-99ac-33eb5546445d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-797m5\" (UID: \"b788786c-b8dd-4a50-99ac-33eb5546445d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5"
Sep 11 14:47:37.891717 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:37.891684    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnwvn\" (UniqueName: \"kubernetes.io/projected/b788786c-b8dd-4a50-99ac-33eb5546445d-kube-api-access-tnwvn\") pod \"kuadrant-operator-controller-manager-5f685c496f-797m5\" (UID: \"b788786c-b8dd-4a50-99ac-33eb5546445d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5"
Sep 11 14:47:37.891879 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:37.891723    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b788786c-b8dd-4a50-99ac-33eb5546445d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-797m5\" (UID: \"b788786c-b8dd-4a50-99ac-33eb5546445d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5"
Sep 11 14:47:37.892083 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:37.892067    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b788786c-b8dd-4a50-99ac-33eb5546445d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-797m5\" (UID: \"b788786c-b8dd-4a50-99ac-33eb5546445d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5"
Sep 11 14:47:37.901968 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:37.901945    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnwvn\" (UniqueName: \"kubernetes.io/projected/b788786c-b8dd-4a50-99ac-33eb5546445d-kube-api-access-tnwvn\") pod \"kuadrant-operator-controller-manager-5f685c496f-797m5\" (UID: \"b788786c-b8dd-4a50-99ac-33eb5546445d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5"
Sep 11 14:47:38.040619 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:38.040589    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5"
Sep 11 14:47:38.319380 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:38.319353    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5"]
Sep 11 14:47:38.321763 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:47:38.321727    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb788786c_b8dd_4a50_99ac_33eb5546445d.slice/crio-ee4804e7ee98b551539a727e30564fd63484c0e8efaadd83995956895a690345 WatchSource:0}: Error finding container ee4804e7ee98b551539a727e30564fd63484c0e8efaadd83995956895a690345: Status 404 returned error can't find the container with id ee4804e7ee98b551539a727e30564fd63484c0e8efaadd83995956895a690345
Sep 11 14:47:38.459801 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:38.459760    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5"]
Sep 11 14:47:38.465687 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:38.465661    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5"]
Sep 11 14:47:38.496036 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:38.496004    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd"]
Sep 11 14:47:38.501809 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:38.501786    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd"
Sep 11 14:47:38.516547 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:38.516516    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd"]
Sep 11 14:47:38.598232 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:38.598156    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a9a07a9-1711-4c92-978b-d72503afd084-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-fbrxd\" (UID: \"0a9a07a9-1711-4c92-978b-d72503afd084\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd"
Sep 11 14:47:38.598232 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:38.598194    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbgsf\" (UniqueName: \"kubernetes.io/projected/0a9a07a9-1711-4c92-978b-d72503afd084-kube-api-access-jbgsf\") pod \"kuadrant-operator-controller-manager-5f685c496f-fbrxd\" (UID: \"0a9a07a9-1711-4c92-978b-d72503afd084\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd"
Sep 11 14:47:38.698964 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:38.698928    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a9a07a9-1711-4c92-978b-d72503afd084-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-fbrxd\" (UID: \"0a9a07a9-1711-4c92-978b-d72503afd084\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd"
Sep 11 14:47:38.698964 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:38.698966    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbgsf\" (UniqueName: \"kubernetes.io/projected/0a9a07a9-1711-4c92-978b-d72503afd084-kube-api-access-jbgsf\") pod \"kuadrant-operator-controller-manager-5f685c496f-fbrxd\" (UID: \"0a9a07a9-1711-4c92-978b-d72503afd084\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd"
Sep 11 14:47:38.699366 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:38.699347    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a9a07a9-1711-4c92-978b-d72503afd084-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-fbrxd\" (UID: \"0a9a07a9-1711-4c92-978b-d72503afd084\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd"
Sep 11 14:47:38.717776 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:38.717748    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbgsf\" (UniqueName: \"kubernetes.io/projected/0a9a07a9-1711-4c92-978b-d72503afd084-kube-api-access-jbgsf\") pod \"kuadrant-operator-controller-manager-5f685c496f-fbrxd\" (UID: \"0a9a07a9-1711-4c92-978b-d72503afd084\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd"
Sep 11 14:47:38.813924 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:38.813894    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd"
Sep 11 14:47:39.099070 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:39.099020    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd"]
Sep 11 14:47:39.101943 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:47:39.101912    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9a07a9_1711_4c92_978b_d72503afd084.slice/crio-5fed14da005854086397c45a1dba9ddcd763c2d1ce17b7c40d3ebe8ba2a44e0d WatchSource:0}: Error finding container 5fed14da005854086397c45a1dba9ddcd763c2d1ce17b7c40d3ebe8ba2a44e0d: Status 404 returned error can't find the container with id 5fed14da005854086397c45a1dba9ddcd763c2d1ce17b7c40d3ebe8ba2a44e0d
Sep 11 14:47:39.717175 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:39.717131    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd" event={"ID":"0a9a07a9-1711-4c92-978b-d72503afd084","Type":"ContainerStarted","Data":"5fed14da005854086397c45a1dba9ddcd763c2d1ce17b7c40d3ebe8ba2a44e0d"}
Sep 11 14:47:42.734396 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:42.734351    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5" podUID="b788786c-b8dd-4a50-99ac-33eb5546445d" containerName="manager" containerID="cri-o://b62344cea3a6a2d4173879431f8fb082de87ec14f5ccdbdb2948a747669b7a6b" gracePeriod=2
Sep 11 14:47:42.736139 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:42.736104    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd" event={"ID":"0a9a07a9-1711-4c92-978b-d72503afd084","Type":"ContainerStarted","Data":"dae342ed5e10351363b0c11a82077ec527d6cbb93c485f9eac29480e7acda142"}
Sep 11 14:47:42.736258 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:42.736243    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd"
Sep 11 14:47:42.736862 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:47:42.736839    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-797m5\" is forbidden: User \"system:node:ip-10-0-133-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-126.ec2.internal' and this object" podUID="b788786c-b8dd-4a50-99ac-33eb5546445d" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5"
Sep 11 14:47:42.739098 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:47:42.738975    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-797m5\" is forbidden: User \"system:node:ip-10-0-133-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-126.ec2.internal' and this object" podUID="b788786c-b8dd-4a50-99ac-33eb5546445d" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5"
Sep 11 14:47:42.762113 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:42.762071    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd" podStartSLOduration=1.657280739 podStartE2EDuration="4.762057024s" totalImagesPullingTime="3.104776285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:47:42.759797822 +0000 UTC m=+442.548062132" watchObservedRunningTime="2026-09-11 14:47:42.762057024 +0000 UTC m=+442.550321322"
Sep 11 14:47:42.969041 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:42.969015    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5"
Sep 11 14:47:42.971576 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:47:42.971551    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-797m5\" is forbidden: User \"system:node:ip-10-0-133-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-126.ec2.internal' and this object" podUID="b788786c-b8dd-4a50-99ac-33eb5546445d" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5"
Sep 11 14:47:43.038317 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:43.038293    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnwvn\" (UniqueName: \"kubernetes.io/projected/b788786c-b8dd-4a50-99ac-33eb5546445d-kube-api-access-tnwvn\") pod \"b788786c-b8dd-4a50-99ac-33eb5546445d\" (UID: \"b788786c-b8dd-4a50-99ac-33eb5546445d\") "
Sep 11 14:47:43.038450 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:43.038336    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b788786c-b8dd-4a50-99ac-33eb5546445d-extensions-socket-volume\") pod \"b788786c-b8dd-4a50-99ac-33eb5546445d\" (UID: \"b788786c-b8dd-4a50-99ac-33eb5546445d\") "
Sep 11 14:47:43.038628 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:43.038603    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b788786c-b8dd-4a50-99ac-33eb5546445d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b788786c-b8dd-4a50-99ac-33eb5546445d" (UID: "b788786c-b8dd-4a50-99ac-33eb5546445d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:47:43.040367 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:43.040343    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b788786c-b8dd-4a50-99ac-33eb5546445d-kube-api-access-tnwvn" (OuterVolumeSpecName: "kube-api-access-tnwvn") pod "b788786c-b8dd-4a50-99ac-33eb5546445d" (UID: "b788786c-b8dd-4a50-99ac-33eb5546445d"). InnerVolumeSpecName "kube-api-access-tnwvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:47:43.138860 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:43.138811    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnwvn\" (UniqueName: \"kubernetes.io/projected/b788786c-b8dd-4a50-99ac-33eb5546445d-kube-api-access-tnwvn\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:47:43.138860 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:43.138853    2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b788786c-b8dd-4a50-99ac-33eb5546445d-extensions-socket-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:47:43.742293 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:43.742255    2568 generic.go:356] "Generic (PLEG): container finished" podID="b788786c-b8dd-4a50-99ac-33eb5546445d" containerID="b62344cea3a6a2d4173879431f8fb082de87ec14f5ccdbdb2948a747669b7a6b" exitCode=2
Sep 11 14:47:43.742744 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:43.742302    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5"
Sep 11 14:47:43.742744 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:43.742353    2568 scope.go:117] "RemoveContainer" containerID="b62344cea3a6a2d4173879431f8fb082de87ec14f5ccdbdb2948a747669b7a6b"
Sep 11 14:47:43.744985 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:47:43.744953    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-797m5\" is forbidden: User \"system:node:ip-10-0-133-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-126.ec2.internal' and this object" podUID="b788786c-b8dd-4a50-99ac-33eb5546445d" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5"
Sep 11 14:47:43.752496 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:43.752478    2568 scope.go:117] "RemoveContainer" containerID="b62344cea3a6a2d4173879431f8fb082de87ec14f5ccdbdb2948a747669b7a6b"
Sep 11 14:47:43.752751 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:47:43.752735    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b62344cea3a6a2d4173879431f8fb082de87ec14f5ccdbdb2948a747669b7a6b\": container with ID starting with b62344cea3a6a2d4173879431f8fb082de87ec14f5ccdbdb2948a747669b7a6b not found: ID does not exist" containerID="b62344cea3a6a2d4173879431f8fb082de87ec14f5ccdbdb2948a747669b7a6b"
Sep 11 14:47:43.752810 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:43.752763    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b62344cea3a6a2d4173879431f8fb082de87ec14f5ccdbdb2948a747669b7a6b"} err="failed to get container status \"b62344cea3a6a2d4173879431f8fb082de87ec14f5ccdbdb2948a747669b7a6b\": rpc error: code = NotFound desc = could not find container \"b62344cea3a6a2d4173879431f8fb082de87ec14f5ccdbdb2948a747669b7a6b\": container with ID starting with b62344cea3a6a2d4173879431f8fb082de87ec14f5ccdbdb2948a747669b7a6b not found: ID does not exist"
Sep 11 14:47:43.754625 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:47:43.754605    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-797m5\" is forbidden: User \"system:node:ip-10-0-133-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-126.ec2.internal' and this object" podUID="b788786c-b8dd-4a50-99ac-33eb5546445d" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-797m5"
Sep 11 14:47:44.661209 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:44.661177    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b788786c-b8dd-4a50-99ac-33eb5546445d" path="/var/lib/kubelet/pods/b788786c-b8dd-4a50-99ac-33eb5546445d/volumes"
Sep 11 14:47:53.744927 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:47:53.744895    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd"
Sep 11 14:48:06.414194 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:06.414156    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd"]
Sep 11 14:48:06.414577 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:06.414348    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd" podUID="0a9a07a9-1711-4c92-978b-d72503afd084" containerName="manager" containerID="cri-o://dae342ed5e10351363b0c11a82077ec527d6cbb93c485f9eac29480e7acda142" gracePeriod=10
Sep 11 14:48:06.663948 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:06.663921    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd"
Sep 11 14:48:06.735885 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:06.735788    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbgsf\" (UniqueName: \"kubernetes.io/projected/0a9a07a9-1711-4c92-978b-d72503afd084-kube-api-access-jbgsf\") pod \"0a9a07a9-1711-4c92-978b-d72503afd084\" (UID: \"0a9a07a9-1711-4c92-978b-d72503afd084\") "
Sep 11 14:48:06.736025 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:06.735905    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a9a07a9-1711-4c92-978b-d72503afd084-extensions-socket-volume\") pod \"0a9a07a9-1711-4c92-978b-d72503afd084\" (UID: \"0a9a07a9-1711-4c92-978b-d72503afd084\") "
Sep 11 14:48:06.736397 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:06.736367    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9a07a9-1711-4c92-978b-d72503afd084-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0a9a07a9-1711-4c92-978b-d72503afd084" (UID: "0a9a07a9-1711-4c92-978b-d72503afd084"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:48:06.737869 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:06.737846    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9a07a9-1711-4c92-978b-d72503afd084-kube-api-access-jbgsf" (OuterVolumeSpecName: "kube-api-access-jbgsf") pod "0a9a07a9-1711-4c92-978b-d72503afd084" (UID: "0a9a07a9-1711-4c92-978b-d72503afd084"). InnerVolumeSpecName "kube-api-access-jbgsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:48:06.837092 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:06.837058    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbgsf\" (UniqueName: \"kubernetes.io/projected/0a9a07a9-1711-4c92-978b-d72503afd084-kube-api-access-jbgsf\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:48:06.837092 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:06.837084    2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a9a07a9-1711-4c92-978b-d72503afd084-extensions-socket-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:48:06.850305 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:06.850268    2568 generic.go:356] "Generic (PLEG): container finished" podID="0a9a07a9-1711-4c92-978b-d72503afd084" containerID="dae342ed5e10351363b0c11a82077ec527d6cbb93c485f9eac29480e7acda142" exitCode=0
Sep 11 14:48:06.850420 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:06.850332    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd"
Sep 11 14:48:06.850420 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:06.850346    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd" event={"ID":"0a9a07a9-1711-4c92-978b-d72503afd084","Type":"ContainerDied","Data":"dae342ed5e10351363b0c11a82077ec527d6cbb93c485f9eac29480e7acda142"}
Sep 11 14:48:06.850488 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:06.850427    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd" event={"ID":"0a9a07a9-1711-4c92-978b-d72503afd084","Type":"ContainerDied","Data":"5fed14da005854086397c45a1dba9ddcd763c2d1ce17b7c40d3ebe8ba2a44e0d"}
Sep 11 14:48:06.850488 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:06.850454    2568 scope.go:117] "RemoveContainer" containerID="dae342ed5e10351363b0c11a82077ec527d6cbb93c485f9eac29480e7acda142"
Sep 11 14:48:06.859144 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:06.859124    2568 scope.go:117] "RemoveContainer" containerID="dae342ed5e10351363b0c11a82077ec527d6cbb93c485f9eac29480e7acda142"
Sep 11 14:48:06.859393 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:48:06.859373    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae342ed5e10351363b0c11a82077ec527d6cbb93c485f9eac29480e7acda142\": container with ID starting with dae342ed5e10351363b0c11a82077ec527d6cbb93c485f9eac29480e7acda142 not found: ID does not exist" containerID="dae342ed5e10351363b0c11a82077ec527d6cbb93c485f9eac29480e7acda142"
Sep 11 14:48:06.859437 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:06.859402    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae342ed5e10351363b0c11a82077ec527d6cbb93c485f9eac29480e7acda142"} err="failed to get container status \"dae342ed5e10351363b0c11a82077ec527d6cbb93c485f9eac29480e7acda142\": rpc error: code = NotFound desc = could not find container \"dae342ed5e10351363b0c11a82077ec527d6cbb93c485f9eac29480e7acda142\": container with ID starting with dae342ed5e10351363b0c11a82077ec527d6cbb93c485f9eac29480e7acda142 not found: ID does not exist"
Sep 11 14:48:06.873362 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:06.873333    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd"]
Sep 11 14:48:06.882698 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:06.882670    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fbrxd"]
Sep 11 14:48:08.661000 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:08.660969    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a9a07a9-1711-4c92-978b-d72503afd084" path="/var/lib/kubelet/pods/0a9a07a9-1711-4c92-978b-d72503afd084/volumes"
Sep 11 14:48:26.161140 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.161105    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-df4h5"]
Sep 11 14:48:26.161505 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.161487    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a9a07a9-1711-4c92-978b-d72503afd084" containerName="manager"
Sep 11 14:48:26.161505 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.161500    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a9a07a9-1711-4c92-978b-d72503afd084" containerName="manager"
Sep 11 14:48:26.161582 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.161511    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b788786c-b8dd-4a50-99ac-33eb5546445d" containerName="manager"
Sep 11 14:48:26.161582 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.161517    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b788786c-b8dd-4a50-99ac-33eb5546445d" containerName="manager"
Sep 11 14:48:26.161645 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.161593    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b788786c-b8dd-4a50-99ac-33eb5546445d" containerName="manager"
Sep 11 14:48:26.161645 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.161603    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a9a07a9-1711-4c92-978b-d72503afd084" containerName="manager"
Sep 11 14:48:26.164961 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.164944    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-df4h5"
Sep 11 14:48:26.167421 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.167392    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 14:48:26.167573 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.167554    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6xbxv\""
Sep 11 14:48:26.175057 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.175034    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-df4h5"]
Sep 11 14:48:26.208121 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.208096    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xq69\" (UniqueName: \"kubernetes.io/projected/40ca3d90-007d-403d-bda4-0acdf2f0f4aa-kube-api-access-9xq69\") pod \"limitador-limitador-8588749977-df4h5\" (UID: \"40ca3d90-007d-403d-bda4-0acdf2f0f4aa\") " pod="kuadrant-system/limitador-limitador-8588749977-df4h5"
Sep 11 14:48:26.208259 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.208145    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40ca3d90-007d-403d-bda4-0acdf2f0f4aa-config-file\") pod \"limitador-limitador-8588749977-df4h5\" (UID: \"40ca3d90-007d-403d-bda4-0acdf2f0f4aa\") " pod="kuadrant-system/limitador-limitador-8588749977-df4h5"
Sep 11 14:48:26.257293 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.257259    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-df4h5"]
Sep 11 14:48:26.308544 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.308511    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xq69\" (UniqueName: \"kubernetes.io/projected/40ca3d90-007d-403d-bda4-0acdf2f0f4aa-kube-api-access-9xq69\") pod \"limitador-limitador-8588749977-df4h5\" (UID: \"40ca3d90-007d-403d-bda4-0acdf2f0f4aa\") " pod="kuadrant-system/limitador-limitador-8588749977-df4h5"
Sep 11 14:48:26.308705 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.308563    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40ca3d90-007d-403d-bda4-0acdf2f0f4aa-config-file\") pod \"limitador-limitador-8588749977-df4h5\" (UID: \"40ca3d90-007d-403d-bda4-0acdf2f0f4aa\") " pod="kuadrant-system/limitador-limitador-8588749977-df4h5"
Sep 11 14:48:26.309131 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.309110    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40ca3d90-007d-403d-bda4-0acdf2f0f4aa-config-file\") pod \"limitador-limitador-8588749977-df4h5\" (UID: \"40ca3d90-007d-403d-bda4-0acdf2f0f4aa\") " pod="kuadrant-system/limitador-limitador-8588749977-df4h5"
Sep 11 14:48:26.317392 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.317364    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xq69\" (UniqueName: \"kubernetes.io/projected/40ca3d90-007d-403d-bda4-0acdf2f0f4aa-kube-api-access-9xq69\") pod \"limitador-limitador-8588749977-df4h5\" (UID: \"40ca3d90-007d-403d-bda4-0acdf2f0f4aa\") " pod="kuadrant-system/limitador-limitador-8588749977-df4h5"
Sep 11 14:48:26.476279 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.476204    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-df4h5"
Sep 11 14:48:26.754979 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.754955    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-df4h5"]
Sep 11 14:48:26.757863 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:48:26.757833    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ca3d90_007d_403d_bda4_0acdf2f0f4aa.slice/crio-cc4a5d11c2ea33cf1571a2ca7914947f7d7864cb1b7a86dc60e6a9e54cb9cb7a WatchSource:0}: Error finding container cc4a5d11c2ea33cf1571a2ca7914947f7d7864cb1b7a86dc60e6a9e54cb9cb7a: Status 404 returned error can't find the container with id cc4a5d11c2ea33cf1571a2ca7914947f7d7864cb1b7a86dc60e6a9e54cb9cb7a
Sep 11 14:48:26.943443 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:26.943407    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-df4h5" event={"ID":"40ca3d90-007d-403d-bda4-0acdf2f0f4aa","Type":"ContainerStarted","Data":"cc4a5d11c2ea33cf1571a2ca7914947f7d7864cb1b7a86dc60e6a9e54cb9cb7a"}
Sep 11 14:48:28.953619 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:28.953580    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-df4h5" event={"ID":"40ca3d90-007d-403d-bda4-0acdf2f0f4aa","Type":"ContainerStarted","Data":"235d988399712f09685dac50fe3b377e2dcc6e647889c88fbcf1041078cbbfe9"}
Sep 11 14:48:28.954096 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:28.953641    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-df4h5"
Sep 11 14:48:28.973352 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:28.973306    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-df4h5" podStartSLOduration=1.7905397079999998 podStartE2EDuration="2.973292427s" totalImagesPullingTime="1.182752719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:48:28.971591289 +0000 UTC m=+488.759855588" watchObservedRunningTime="2026-09-11 14:48:28.973292427 +0000 UTC m=+488.761556725"
Sep 11 14:48:39.959217 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:39.959183    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-df4h5"
Sep 11 14:48:40.572162 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:40.572124    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-df4h5"]
Sep 11 14:48:40.572364 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:40.572316    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-df4h5" podUID="40ca3d90-007d-403d-bda4-0acdf2f0f4aa" containerName="limitador" containerID="cri-o://235d988399712f09685dac50fe3b377e2dcc6e647889c88fbcf1041078cbbfe9" gracePeriod=30
Sep 11 14:48:41.007443 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:41.007401    2568 generic.go:356] "Generic (PLEG): container finished" podID="40ca3d90-007d-403d-bda4-0acdf2f0f4aa" containerID="235d988399712f09685dac50fe3b377e2dcc6e647889c88fbcf1041078cbbfe9" exitCode=0
Sep 11 14:48:41.007769 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:41.007465    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-df4h5" event={"ID":"40ca3d90-007d-403d-bda4-0acdf2f0f4aa","Type":"ContainerDied","Data":"235d988399712f09685dac50fe3b377e2dcc6e647889c88fbcf1041078cbbfe9"}
Sep 11 14:48:41.117242 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:41.117219    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-df4h5"
Sep 11 14:48:41.232078 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:41.231991    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40ca3d90-007d-403d-bda4-0acdf2f0f4aa-config-file\") pod \"40ca3d90-007d-403d-bda4-0acdf2f0f4aa\" (UID: \"40ca3d90-007d-403d-bda4-0acdf2f0f4aa\") "
Sep 11 14:48:41.232078 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:41.232077    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xq69\" (UniqueName: \"kubernetes.io/projected/40ca3d90-007d-403d-bda4-0acdf2f0f4aa-kube-api-access-9xq69\") pod \"40ca3d90-007d-403d-bda4-0acdf2f0f4aa\" (UID: \"40ca3d90-007d-403d-bda4-0acdf2f0f4aa\") "
Sep 11 14:48:41.232357 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:41.232330    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40ca3d90-007d-403d-bda4-0acdf2f0f4aa-config-file" (OuterVolumeSpecName: "config-file") pod "40ca3d90-007d-403d-bda4-0acdf2f0f4aa" (UID: "40ca3d90-007d-403d-bda4-0acdf2f0f4aa"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:48:41.234208 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:41.234183    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ca3d90-007d-403d-bda4-0acdf2f0f4aa-kube-api-access-9xq69" (OuterVolumeSpecName: "kube-api-access-9xq69") pod "40ca3d90-007d-403d-bda4-0acdf2f0f4aa" (UID: "40ca3d90-007d-403d-bda4-0acdf2f0f4aa"). InnerVolumeSpecName "kube-api-access-9xq69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:48:41.332905 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:41.332872    2568 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40ca3d90-007d-403d-bda4-0acdf2f0f4aa-config-file\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:48:41.332905 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:41.332900    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xq69\" (UniqueName: \"kubernetes.io/projected/40ca3d90-007d-403d-bda4-0acdf2f0f4aa-kube-api-access-9xq69\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:48:42.014366 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:42.014328    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-df4h5" event={"ID":"40ca3d90-007d-403d-bda4-0acdf2f0f4aa","Type":"ContainerDied","Data":"cc4a5d11c2ea33cf1571a2ca7914947f7d7864cb1b7a86dc60e6a9e54cb9cb7a"}
Sep 11 14:48:42.014366 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:42.014366    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-df4h5"
Sep 11 14:48:42.014848 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:42.014416    2568 scope.go:117] "RemoveContainer" containerID="235d988399712f09685dac50fe3b377e2dcc6e647889c88fbcf1041078cbbfe9"
Sep 11 14:48:42.038118 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:42.038091    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-df4h5"]
Sep 11 14:48:42.041711 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:42.041691    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-df4h5"]
Sep 11 14:48:42.660957 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:42.660929    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40ca3d90-007d-403d-bda4-0acdf2f0f4aa" path="/var/lib/kubelet/pods/40ca3d90-007d-403d-bda4-0acdf2f0f4aa/volumes"
Sep 11 14:48:44.706765 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:44.706733    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nb9vs"]
Sep 11 14:48:44.707137 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:44.707078    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40ca3d90-007d-403d-bda4-0acdf2f0f4aa" containerName="limitador"
Sep 11 14:48:44.707137 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:44.707088    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="40ca3d90-007d-403d-bda4-0acdf2f0f4aa" containerName="limitador"
Sep 11 14:48:44.707217 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:44.707147    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="40ca3d90-007d-403d-bda4-0acdf2f0f4aa" containerName="limitador"
Sep 11 14:48:44.710102 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:44.710085    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nb9vs"
Sep 11 14:48:44.714800 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:48:44.714772    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"postgres-creds\" is forbidden: User \"system:node:ip-10-0-133-126.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-133-126.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" type="*v1.Secret"
Sep 11 14:48:44.717691 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:44.717671    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 14:48:44.719864 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:48:44.719838    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-133-126.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-133-126.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 11 14:48:44.719864 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:44.719848    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-dqs5f\""
Sep 11 14:48:44.729725 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:44.729703    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nb9vs"]
Sep 11 14:48:44.759677 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:44.759651    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1a473e00-62c2-4d56-acab-60d7f3026db4-data\") pod \"postgres-69fb4594cf-nb9vs\" (UID: \"1a473e00-62c2-4d56-acab-60d7f3026db4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nb9vs"
Sep 11 14:48:44.759760 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:44.759708    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27tqm\" (UniqueName: \"kubernetes.io/projected/1a473e00-62c2-4d56-acab-60d7f3026db4-kube-api-access-27tqm\") pod \"postgres-69fb4594cf-nb9vs\" (UID: \"1a473e00-62c2-4d56-acab-60d7f3026db4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nb9vs"
Sep 11 14:48:44.860544 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:44.860514    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27tqm\" (UniqueName: \"kubernetes.io/projected/1a473e00-62c2-4d56-acab-60d7f3026db4-kube-api-access-27tqm\") pod \"postgres-69fb4594cf-nb9vs\" (UID: \"1a473e00-62c2-4d56-acab-60d7f3026db4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nb9vs"
Sep 11 14:48:44.860704 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:44.860593    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1a473e00-62c2-4d56-acab-60d7f3026db4-data\") pod \"postgres-69fb4594cf-nb9vs\" (UID: \"1a473e00-62c2-4d56-acab-60d7f3026db4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nb9vs"
Sep 11 14:48:44.861016 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:44.860999    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1a473e00-62c2-4d56-acab-60d7f3026db4-data\") pod \"postgres-69fb4594cf-nb9vs\" (UID: \"1a473e00-62c2-4d56-acab-60d7f3026db4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nb9vs"
Sep 11 14:48:45.660547 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:45.660515    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 11 14:48:45.772543 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:45.772514    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:48:45.780958 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:45.780934    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27tqm\" (UniqueName: \"kubernetes.io/projected/1a473e00-62c2-4d56-acab-60d7f3026db4-kube-api-access-27tqm\") pod \"postgres-69fb4594cf-nb9vs\" (UID: \"1a473e00-62c2-4d56-acab-60d7f3026db4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nb9vs"
Sep 11 14:48:45.919736 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:45.919665    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nb9vs"
Sep 11 14:48:46.190528 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:46.190377    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nb9vs"]
Sep 11 14:48:46.192579 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:48:46.192552    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a473e00_62c2_4d56_acab_60d7f3026db4.slice/crio-0787cf1b0ed9dc1594e1a8ae7b493e496affa45a8304dbebff414fa7796da7f7 WatchSource:0}: Error finding container 0787cf1b0ed9dc1594e1a8ae7b493e496affa45a8304dbebff414fa7796da7f7: Status 404 returned error can't find the container with id 0787cf1b0ed9dc1594e1a8ae7b493e496affa45a8304dbebff414fa7796da7f7
Sep 11 14:48:47.038376 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:47.038338    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nb9vs" event={"ID":"1a473e00-62c2-4d56-acab-60d7f3026db4","Type":"ContainerStarted","Data":"0787cf1b0ed9dc1594e1a8ae7b493e496affa45a8304dbebff414fa7796da7f7"}
Sep 11 14:48:52.064030 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:52.063997    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nb9vs" event={"ID":"1a473e00-62c2-4d56-acab-60d7f3026db4","Type":"ContainerStarted","Data":"8a9d790b87a41814fd16941c28a7f489b20816657ee106112fc7deb1fbc26aec"}
Sep 11 14:48:52.064416 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:52.064117    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nb9vs"
Sep 11 14:48:52.081531 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:52.081487    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nb9vs" podStartSLOduration=3.028796917 podStartE2EDuration="8.081473018s" totalImagesPullingTime="5.052676101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:48:52.081093608 +0000 UTC m=+511.869357907" watchObservedRunningTime="2026-09-11 14:48:52.081473018 +0000 UTC m=+511.869737318"
Sep 11 14:48:58.096893 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:48:58.096857    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nb9vs"
Sep 11 14:49:05.996712 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:05.996679    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8679c74fb7-r2gxp"]
Sep 11 14:49:06.021662 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.021631    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8679c74fb7-r2gxp"]
Sep 11 14:49:06.021812 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.021740    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8679c74fb7-r2gxp"
Sep 11 14:49:06.024444 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.024419    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 14:49:06.024589 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.024540    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 14:49:06.024663 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.024642    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sd75p\""
Sep 11 14:49:06.024785 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.024642    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 14:49:06.152277 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.152240    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vgc7\" (UniqueName: \"kubernetes.io/projected/f04d2427-f7c9-4145-a9f5-a0e48a6d02e0-kube-api-access-9vgc7\") pod \"maas-controller-8679c74fb7-r2gxp\" (UID: \"f04d2427-f7c9-4145-a9f5-a0e48a6d02e0\") " pod="opendatahub/maas-controller-8679c74fb7-r2gxp"
Sep 11 14:49:06.152463 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.152302    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f04d2427-f7c9-4145-a9f5-a0e48a6d02e0-webhook-cert\") pod \"maas-controller-8679c74fb7-r2gxp\" (UID: \"f04d2427-f7c9-4145-a9f5-a0e48a6d02e0\") " pod="opendatahub/maas-controller-8679c74fb7-r2gxp"
Sep 11 14:49:06.152463 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.152386    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f04d2427-f7c9-4145-a9f5-a0e48a6d02e0-metrics-certs\") pod \"maas-controller-8679c74fb7-r2gxp\" (UID: \"f04d2427-f7c9-4145-a9f5-a0e48a6d02e0\") " pod="opendatahub/maas-controller-8679c74fb7-r2gxp"
Sep 11 14:49:06.184091 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.184060    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bdb5c7ff-jz27x"]
Sep 11 14:49:06.187515 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.187499    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bdb5c7ff-jz27x"
Sep 11 14:49:06.199535 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.199513    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bdb5c7ff-jz27x"]
Sep 11 14:49:06.253299 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.253234    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vgc7\" (UniqueName: \"kubernetes.io/projected/f04d2427-f7c9-4145-a9f5-a0e48a6d02e0-kube-api-access-9vgc7\") pod \"maas-controller-8679c74fb7-r2gxp\" (UID: \"f04d2427-f7c9-4145-a9f5-a0e48a6d02e0\") " pod="opendatahub/maas-controller-8679c74fb7-r2gxp"
Sep 11 14:49:06.253299 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.253283    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f04d2427-f7c9-4145-a9f5-a0e48a6d02e0-webhook-cert\") pod \"maas-controller-8679c74fb7-r2gxp\" (UID: \"f04d2427-f7c9-4145-a9f5-a0e48a6d02e0\") " pod="opendatahub/maas-controller-8679c74fb7-r2gxp"
Sep 11 14:49:06.253466 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.253323    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f04d2427-f7c9-4145-a9f5-a0e48a6d02e0-metrics-certs\") pod \"maas-controller-8679c74fb7-r2gxp\" (UID: \"f04d2427-f7c9-4145-a9f5-a0e48a6d02e0\") " pod="opendatahub/maas-controller-8679c74fb7-r2gxp"
Sep 11 14:49:06.255743 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.255721    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f04d2427-f7c9-4145-a9f5-a0e48a6d02e0-webhook-cert\") pod \"maas-controller-8679c74fb7-r2gxp\" (UID: \"f04d2427-f7c9-4145-a9f5-a0e48a6d02e0\") " pod="opendatahub/maas-controller-8679c74fb7-r2gxp"
Sep 11 14:49:06.255858 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.255772    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f04d2427-f7c9-4145-a9f5-a0e48a6d02e0-metrics-certs\") pod \"maas-controller-8679c74fb7-r2gxp\" (UID: \"f04d2427-f7c9-4145-a9f5-a0e48a6d02e0\") " pod="opendatahub/maas-controller-8679c74fb7-r2gxp"
Sep 11 14:49:06.262171 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.262144    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vgc7\" (UniqueName: \"kubernetes.io/projected/f04d2427-f7c9-4145-a9f5-a0e48a6d02e0-kube-api-access-9vgc7\") pod \"maas-controller-8679c74fb7-r2gxp\" (UID: \"f04d2427-f7c9-4145-a9f5-a0e48a6d02e0\") " pod="opendatahub/maas-controller-8679c74fb7-r2gxp"
Sep 11 14:49:06.333980 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.333947    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8679c74fb7-r2gxp"
Sep 11 14:49:06.354511 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.354487    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a191532-c062-4aed-ab16-771a4bb32f16-metrics-certs\") pod \"maas-controller-5bdb5c7ff-jz27x\" (UID: \"6a191532-c062-4aed-ab16-771a4bb32f16\") " pod="opendatahub/maas-controller-5bdb5c7ff-jz27x"
Sep 11 14:49:06.354511 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.354518    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tmtc\" (UniqueName: \"kubernetes.io/projected/6a191532-c062-4aed-ab16-771a4bb32f16-kube-api-access-8tmtc\") pod \"maas-controller-5bdb5c7ff-jz27x\" (UID: \"6a191532-c062-4aed-ab16-771a4bb32f16\") " pod="opendatahub/maas-controller-5bdb5c7ff-jz27x"
Sep 11 14:49:06.354665 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.354539    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a191532-c062-4aed-ab16-771a4bb32f16-webhook-cert\") pod \"maas-controller-5bdb5c7ff-jz27x\" (UID: \"6a191532-c062-4aed-ab16-771a4bb32f16\") " pod="opendatahub/maas-controller-5bdb5c7ff-jz27x"
Sep 11 14:49:06.455770 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.455740    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a191532-c062-4aed-ab16-771a4bb32f16-metrics-certs\") pod \"maas-controller-5bdb5c7ff-jz27x\" (UID: \"6a191532-c062-4aed-ab16-771a4bb32f16\") " pod="opendatahub/maas-controller-5bdb5c7ff-jz27x"
Sep 11 14:49:06.455936 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.455777    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tmtc\" (UniqueName: \"kubernetes.io/projected/6a191532-c062-4aed-ab16-771a4bb32f16-kube-api-access-8tmtc\") pod \"maas-controller-5bdb5c7ff-jz27x\" (UID: \"6a191532-c062-4aed-ab16-771a4bb32f16\") " pod="opendatahub/maas-controller-5bdb5c7ff-jz27x"
Sep 11 14:49:06.455936 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.455804    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a191532-c062-4aed-ab16-771a4bb32f16-webhook-cert\") pod \"maas-controller-5bdb5c7ff-jz27x\" (UID: \"6a191532-c062-4aed-ab16-771a4bb32f16\") " pod="opendatahub/maas-controller-5bdb5c7ff-jz27x"
Sep 11 14:49:06.458184 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.458160    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a191532-c062-4aed-ab16-771a4bb32f16-metrics-certs\") pod \"maas-controller-5bdb5c7ff-jz27x\" (UID: \"6a191532-c062-4aed-ab16-771a4bb32f16\") " pod="opendatahub/maas-controller-5bdb5c7ff-jz27x"
Sep 11 14:49:06.458273 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.458209    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a191532-c062-4aed-ab16-771a4bb32f16-webhook-cert\") pod \"maas-controller-5bdb5c7ff-jz27x\" (UID: \"6a191532-c062-4aed-ab16-771a4bb32f16\") " pod="opendatahub/maas-controller-5bdb5c7ff-jz27x"
Sep 11 14:49:06.465322 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.465301    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tmtc\" (UniqueName: \"kubernetes.io/projected/6a191532-c062-4aed-ab16-771a4bb32f16-kube-api-access-8tmtc\") pod \"maas-controller-5bdb5c7ff-jz27x\" (UID: \"6a191532-c062-4aed-ab16-771a4bb32f16\") " pod="opendatahub/maas-controller-5bdb5c7ff-jz27x"
Sep 11 14:49:06.498188 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.498170    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bdb5c7ff-jz27x"
Sep 11 14:49:06.605866 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.605761    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8679c74fb7-r2gxp"]
Sep 11 14:49:06.608051 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:49:06.608029    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04d2427_f7c9_4145_a9f5_a0e48a6d02e0.slice/crio-c7400065a9806519190a38da7784572f67d4f44588cabdcddff6c322cf552cbd WatchSource:0}: Error finding container c7400065a9806519190a38da7784572f67d4f44588cabdcddff6c322cf552cbd: Status 404 returned error can't find the container with id c7400065a9806519190a38da7784572f67d4f44588cabdcddff6c322cf552cbd
Sep 11 14:49:06.770971 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:06.770943    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bdb5c7ff-jz27x"]
Sep 11 14:49:06.772783 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:49:06.772760    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a191532_c062_4aed_ab16_771a4bb32f16.slice/crio-a69521207d514ed05d5c2a992f3f05483d87f56e66ec8514f16425e8582304f9 WatchSource:0}: Error finding container a69521207d514ed05d5c2a992f3f05483d87f56e66ec8514f16425e8582304f9: Status 404 returned error can't find the container with id a69521207d514ed05d5c2a992f3f05483d87f56e66ec8514f16425e8582304f9
Sep 11 14:49:07.132602 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:07.132562    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bdb5c7ff-jz27x" event={"ID":"6a191532-c062-4aed-ab16-771a4bb32f16","Type":"ContainerStarted","Data":"a69521207d514ed05d5c2a992f3f05483d87f56e66ec8514f16425e8582304f9"}
Sep 11 14:49:07.133707 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:07.133685    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8679c74fb7-r2gxp" event={"ID":"f04d2427-f7c9-4145-a9f5-a0e48a6d02e0","Type":"ContainerStarted","Data":"c7400065a9806519190a38da7784572f67d4f44588cabdcddff6c322cf552cbd"}
Sep 11 14:49:10.152499 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:10.152462    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bdb5c7ff-jz27x" event={"ID":"6a191532-c062-4aed-ab16-771a4bb32f16","Type":"ContainerStarted","Data":"9d692337f53edaf4e429076d58ff9d96824bf04dad304b45912de520895aad52"}
Sep 11 14:49:10.152928 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:10.152568    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bdb5c7ff-jz27x"
Sep 11 14:49:10.154078 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:10.154054    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8679c74fb7-r2gxp" event={"ID":"f04d2427-f7c9-4145-a9f5-a0e48a6d02e0","Type":"ContainerStarted","Data":"077e6dcbed38aa39caa09d27fbafd1430742b5dc8349a853e1b2006e23e26796"}
Sep 11 14:49:10.154193 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:10.154097    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8679c74fb7-r2gxp"
Sep 11 14:49:10.172130 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:10.172084    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5bdb5c7ff-jz27x" podStartSLOduration=1.632073423 podStartE2EDuration="4.17206987s" totalImagesPullingTime="2.539996447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:49:10.171206908 +0000 UTC m=+529.959471208" watchObservedRunningTime="2026-09-11 14:49:10.17206987 +0000 UTC m=+529.960334169"
Sep 11 14:49:10.193460 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:10.193407    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8679c74fb7-r2gxp" podStartSLOduration=2.498671269 podStartE2EDuration="5.193394257s" totalImagesPullingTime="2.694722988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:49:10.192080769 +0000 UTC m=+529.980345068" watchObservedRunningTime="2026-09-11 14:49:10.193394257 +0000 UTC m=+529.981658558"
Sep 11 14:49:21.161429 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:21.161394    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8679c74fb7-r2gxp"
Sep 11 14:49:21.161837 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:21.161475    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bdb5c7ff-jz27x"
Sep 11 14:49:21.234754 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:21.234715    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8679c74fb7-r2gxp"]
Sep 11 14:49:21.234948 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:21.234922    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-8679c74fb7-r2gxp" podUID="f04d2427-f7c9-4145-a9f5-a0e48a6d02e0" containerName="manager" containerID="cri-o://077e6dcbed38aa39caa09d27fbafd1430742b5dc8349a853e1b2006e23e26796" gracePeriod=30
Sep 11 14:49:21.476979 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:21.476952    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8679c74fb7-r2gxp"
Sep 11 14:49:21.487040 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:21.487019    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f04d2427-f7c9-4145-a9f5-a0e48a6d02e0-metrics-certs\") pod \"f04d2427-f7c9-4145-a9f5-a0e48a6d02e0\" (UID: \"f04d2427-f7c9-4145-a9f5-a0e48a6d02e0\") "
Sep 11 14:49:21.487142 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:21.487074    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vgc7\" (UniqueName: \"kubernetes.io/projected/f04d2427-f7c9-4145-a9f5-a0e48a6d02e0-kube-api-access-9vgc7\") pod \"f04d2427-f7c9-4145-a9f5-a0e48a6d02e0\" (UID: \"f04d2427-f7c9-4145-a9f5-a0e48a6d02e0\") "
Sep 11 14:49:21.487142 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:21.487114    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f04d2427-f7c9-4145-a9f5-a0e48a6d02e0-webhook-cert\") pod \"f04d2427-f7c9-4145-a9f5-a0e48a6d02e0\" (UID: \"f04d2427-f7c9-4145-a9f5-a0e48a6d02e0\") "
Sep 11 14:49:21.489071 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:21.489048    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f04d2427-f7c9-4145-a9f5-a0e48a6d02e0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f04d2427-f7c9-4145-a9f5-a0e48a6d02e0" (UID: "f04d2427-f7c9-4145-a9f5-a0e48a6d02e0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:49:21.489207 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:21.489190    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f04d2427-f7c9-4145-a9f5-a0e48a6d02e0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f04d2427-f7c9-4145-a9f5-a0e48a6d02e0" (UID: "f04d2427-f7c9-4145-a9f5-a0e48a6d02e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:49:21.489260 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:21.489208    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04d2427-f7c9-4145-a9f5-a0e48a6d02e0-kube-api-access-9vgc7" (OuterVolumeSpecName: "kube-api-access-9vgc7") pod "f04d2427-f7c9-4145-a9f5-a0e48a6d02e0" (UID: "f04d2427-f7c9-4145-a9f5-a0e48a6d02e0"). InnerVolumeSpecName "kube-api-access-9vgc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:49:21.588307 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:21.588273    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vgc7\" (UniqueName: \"kubernetes.io/projected/f04d2427-f7c9-4145-a9f5-a0e48a6d02e0-kube-api-access-9vgc7\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:49:21.588307 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:21.588302    2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f04d2427-f7c9-4145-a9f5-a0e48a6d02e0-webhook-cert\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:49:21.588307 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:21.588312    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f04d2427-f7c9-4145-a9f5-a0e48a6d02e0-metrics-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:49:22.211642 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:22.211603    2568 generic.go:356] "Generic (PLEG): container finished" podID="f04d2427-f7c9-4145-a9f5-a0e48a6d02e0" containerID="077e6dcbed38aa39caa09d27fbafd1430742b5dc8349a853e1b2006e23e26796" exitCode=0
Sep 11 14:49:22.212094 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:22.211649    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8679c74fb7-r2gxp"
Sep 11 14:49:22.212094 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:22.211689    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8679c74fb7-r2gxp" event={"ID":"f04d2427-f7c9-4145-a9f5-a0e48a6d02e0","Type":"ContainerDied","Data":"077e6dcbed38aa39caa09d27fbafd1430742b5dc8349a853e1b2006e23e26796"}
Sep 11 14:49:22.212094 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:22.211736    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8679c74fb7-r2gxp" event={"ID":"f04d2427-f7c9-4145-a9f5-a0e48a6d02e0","Type":"ContainerDied","Data":"c7400065a9806519190a38da7784572f67d4f44588cabdcddff6c322cf552cbd"}
Sep 11 14:49:22.212094 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:22.211759    2568 scope.go:117] "RemoveContainer" containerID="077e6dcbed38aa39caa09d27fbafd1430742b5dc8349a853e1b2006e23e26796"
Sep 11 14:49:22.221091 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:22.221077    2568 scope.go:117] "RemoveContainer" containerID="077e6dcbed38aa39caa09d27fbafd1430742b5dc8349a853e1b2006e23e26796"
Sep 11 14:49:22.221320 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:49:22.221306    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077e6dcbed38aa39caa09d27fbafd1430742b5dc8349a853e1b2006e23e26796\": container with ID starting with 077e6dcbed38aa39caa09d27fbafd1430742b5dc8349a853e1b2006e23e26796 not found: ID does not exist" containerID="077e6dcbed38aa39caa09d27fbafd1430742b5dc8349a853e1b2006e23e26796"
Sep 11 14:49:22.221368 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:22.221327    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077e6dcbed38aa39caa09d27fbafd1430742b5dc8349a853e1b2006e23e26796"} err="failed to get container status \"077e6dcbed38aa39caa09d27fbafd1430742b5dc8349a853e1b2006e23e26796\": rpc error: code = NotFound desc = could not find container \"077e6dcbed38aa39caa09d27fbafd1430742b5dc8349a853e1b2006e23e26796\": container with ID starting with 077e6dcbed38aa39caa09d27fbafd1430742b5dc8349a853e1b2006e23e26796 not found: ID does not exist"
Sep 11 14:49:22.233738 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:22.233714    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8679c74fb7-r2gxp"]
Sep 11 14:49:22.237276 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:22.237250    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8679c74fb7-r2gxp"]
Sep 11 14:49:22.662847 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:22.662608    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f04d2427-f7c9-4145-a9f5-a0e48a6d02e0" path="/var/lib/kubelet/pods/f04d2427-f7c9-4145-a9f5-a0e48a6d02e0/volumes"
Sep 11 14:49:44.768106 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:44.768073    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-56956d4668-hh4br"]
Sep 11 14:49:44.768486 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:44.768420    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f04d2427-f7c9-4145-a9f5-a0e48a6d02e0" containerName="manager"
Sep 11 14:49:44.768486 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:44.768431    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f04d2427-f7c9-4145-a9f5-a0e48a6d02e0" containerName="manager"
Sep 11 14:49:44.768571 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:44.768507    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f04d2427-f7c9-4145-a9f5-a0e48a6d02e0" containerName="manager"
Sep 11 14:49:44.771486 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:44.771464    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br"
Sep 11 14:49:44.778356 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:44.778333    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 14:49:44.778520 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:44.778500    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 14:49:44.787021 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:44.786998    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-56956d4668-hh4br"]
Sep 11 14:49:44.878935 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:44.878903    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/802087f4-e8fa-4000-80c4-4c4a3179177d-metrics-certs\") pod \"maas-api-56956d4668-hh4br\" (UID: \"802087f4-e8fa-4000-80c4-4c4a3179177d\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br"
Sep 11 14:49:44.879095 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:44.878945    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms4zf\" (UniqueName: \"kubernetes.io/projected/802087f4-e8fa-4000-80c4-4c4a3179177d-kube-api-access-ms4zf\") pod \"maas-api-56956d4668-hh4br\" (UID: \"802087f4-e8fa-4000-80c4-4c4a3179177d\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br"
Sep 11 14:49:44.879095 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:44.878973    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/802087f4-e8fa-4000-80c4-4c4a3179177d-maas-api-tls\") pod \"maas-api-56956d4668-hh4br\" (UID: \"802087f4-e8fa-4000-80c4-4c4a3179177d\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br"
Sep 11 14:49:44.879095 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:44.879048    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/802087f4-e8fa-4000-80c4-4c4a3179177d-openshift-service-ca\") pod \"maas-api-56956d4668-hh4br\" (UID: \"802087f4-e8fa-4000-80c4-4c4a3179177d\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br"
Sep 11 14:49:44.979756 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:44.979716    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/802087f4-e8fa-4000-80c4-4c4a3179177d-openshift-service-ca\") pod \"maas-api-56956d4668-hh4br\" (UID: \"802087f4-e8fa-4000-80c4-4c4a3179177d\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br"
Sep 11 14:49:44.979940 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:44.979803    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/802087f4-e8fa-4000-80c4-4c4a3179177d-metrics-certs\") pod \"maas-api-56956d4668-hh4br\" (UID: \"802087f4-e8fa-4000-80c4-4c4a3179177d\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br"
Sep 11 14:49:44.979940 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:44.979876    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms4zf\" (UniqueName: \"kubernetes.io/projected/802087f4-e8fa-4000-80c4-4c4a3179177d-kube-api-access-ms4zf\") pod \"maas-api-56956d4668-hh4br\" (UID: \"802087f4-e8fa-4000-80c4-4c4a3179177d\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br"
Sep 11 14:49:44.979940 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:44.979904    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/802087f4-e8fa-4000-80c4-4c4a3179177d-maas-api-tls\") pod \"maas-api-56956d4668-hh4br\" (UID: \"802087f4-e8fa-4000-80c4-4c4a3179177d\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br"
Sep 11 14:49:44.980534 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:44.980507    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/802087f4-e8fa-4000-80c4-4c4a3179177d-openshift-service-ca\") pod \"maas-api-56956d4668-hh4br\" (UID: \"802087f4-e8fa-4000-80c4-4c4a3179177d\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br"
Sep 11 14:49:44.982331 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:44.982312    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/802087f4-e8fa-4000-80c4-4c4a3179177d-metrics-certs\") pod \"maas-api-56956d4668-hh4br\" (UID: \"802087f4-e8fa-4000-80c4-4c4a3179177d\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br"
Sep 11 14:49:44.982417 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:44.982366    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/802087f4-e8fa-4000-80c4-4c4a3179177d-maas-api-tls\") pod \"maas-api-56956d4668-hh4br\" (UID: \"802087f4-e8fa-4000-80c4-4c4a3179177d\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br"
Sep 11 14:49:44.988965 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:44.988945    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms4zf\" (UniqueName: \"kubernetes.io/projected/802087f4-e8fa-4000-80c4-4c4a3179177d-kube-api-access-ms4zf\") pod \"maas-api-56956d4668-hh4br\" (UID: \"802087f4-e8fa-4000-80c4-4c4a3179177d\") " pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br"
Sep 11 14:49:45.081575 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:45.081541    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br"
Sep 11 14:49:45.355414 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:45.355386    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-56956d4668-hh4br"]
Sep 11 14:49:45.357508 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:49:45.357479    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802087f4_e8fa_4000_80c4_4c4a3179177d.slice/crio-cac1e7e9dca89006444d109c9e713c7183d1db9d810fc3efe540c93ffba4a88d WatchSource:0}: Error finding container cac1e7e9dca89006444d109c9e713c7183d1db9d810fc3efe540c93ffba4a88d: Status 404 returned error can't find the container with id cac1e7e9dca89006444d109c9e713c7183d1db9d810fc3efe540c93ffba4a88d
Sep 11 14:49:45.401860 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:45.401829    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-rwvw4"]
Sep 11 14:49:45.406482 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:45.406462    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-rwvw4"
Sep 11 14:49:45.409284 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:45.409263    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 14:49:45.409681 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:45.409662    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4xm7h\""
Sep 11 14:49:45.419401 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:45.419380    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-rwvw4"]
Sep 11 14:49:45.483874 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:45.483842    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk2n9\" (UniqueName: \"kubernetes.io/projected/0bc37496-0e43-4d25-a663-a96dd2e1f64e-kube-api-access-kk2n9\") pod \"payload-processing-5b54b7fbb7-rwvw4\" (UID: \"0bc37496-0e43-4d25-a663-a96dd2e1f64e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-rwvw4"
Sep 11 14:49:45.484013 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:45.483887    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc37496-0e43-4d25-a663-a96dd2e1f64e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-rwvw4\" (UID: \"0bc37496-0e43-4d25-a663-a96dd2e1f64e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-rwvw4"
Sep 11 14:49:45.585257 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:45.585224    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk2n9\" (UniqueName: \"kubernetes.io/projected/0bc37496-0e43-4d25-a663-a96dd2e1f64e-kube-api-access-kk2n9\") pod \"payload-processing-5b54b7fbb7-rwvw4\" (UID: \"0bc37496-0e43-4d25-a663-a96dd2e1f64e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-rwvw4"
Sep 11 14:49:45.585414 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:45.585264    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc37496-0e43-4d25-a663-a96dd2e1f64e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-rwvw4\" (UID: \"0bc37496-0e43-4d25-a663-a96dd2e1f64e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-rwvw4"
Sep 11 14:49:45.586061 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:45.586042    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc37496-0e43-4d25-a663-a96dd2e1f64e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-rwvw4\" (UID: \"0bc37496-0e43-4d25-a663-a96dd2e1f64e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-rwvw4"
Sep 11 14:49:45.598908 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:45.598883    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk2n9\" (UniqueName: \"kubernetes.io/projected/0bc37496-0e43-4d25-a663-a96dd2e1f64e-kube-api-access-kk2n9\") pod \"payload-processing-5b54b7fbb7-rwvw4\" (UID: \"0bc37496-0e43-4d25-a663-a96dd2e1f64e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-rwvw4"
Sep 11 14:49:45.716411 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:45.716326    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-rwvw4"
Sep 11 14:49:45.992612 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:45.992585    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-rwvw4"]
Sep 11 14:49:45.994544 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:49:45.994512    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc37496_0e43_4d25_a663_a96dd2e1f64e.slice/crio-a610dcd6d6ac61ca4a376a3dcd3c6434ea74025696a4e27609248511deef9059 WatchSource:0}: Error finding container a610dcd6d6ac61ca4a376a3dcd3c6434ea74025696a4e27609248511deef9059: Status 404 returned error can't find the container with id a610dcd6d6ac61ca4a376a3dcd3c6434ea74025696a4e27609248511deef9059
Sep 11 14:49:46.324887 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:46.324843    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br" event={"ID":"802087f4-e8fa-4000-80c4-4c4a3179177d","Type":"ContainerStarted","Data":"cac1e7e9dca89006444d109c9e713c7183d1db9d810fc3efe540c93ffba4a88d"}
Sep 11 14:49:46.326135 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:46.326106    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-rwvw4" event={"ID":"0bc37496-0e43-4d25-a663-a96dd2e1f64e","Type":"ContainerStarted","Data":"a610dcd6d6ac61ca4a376a3dcd3c6434ea74025696a4e27609248511deef9059"}
Sep 11 14:49:48.337411 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:48.337369    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br" event={"ID":"802087f4-e8fa-4000-80c4-4c4a3179177d","Type":"ContainerStarted","Data":"6b144c06bc3113342fa33b8f6aeeab6101d2baf54917e07ce77056c62745738b"}
Sep 11 14:49:48.337885 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:48.337448    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br"
Sep 11 14:49:48.344928 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:48.344897    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-rwvw4" event={"ID":"0bc37496-0e43-4d25-a663-a96dd2e1f64e","Type":"ContainerStarted","Data":"f30c69c3972d335a8d6973c09d4b802a94ad12af6f1d0779b37df23c01be58e5"}
Sep 11 14:49:48.345186 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:48.345168    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-rwvw4"
Sep 11 14:49:48.357004 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:48.356958    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br" podStartSLOduration=1.8704642379999998 podStartE2EDuration="4.356942893s" totalImagesPullingTime="2.486478655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:49:48.35686207 +0000 UTC m=+568.145126372" watchObservedRunningTime="2026-09-11 14:49:48.356942893 +0000 UTC m=+568.145207193"
Sep 11 14:49:54.355611 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:54.355579    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br"
Sep 11 14:49:54.376336 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:54.376288    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-rwvw4" podStartSLOduration=7.52153702 podStartE2EDuration="9.376274274s" totalImagesPullingTime="1.854737254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:49:48.37404133 +0000 UTC m=+568.162305632" watchObservedRunningTime="2026-09-11 14:49:54.376274274 +0000 UTC m=+574.164538574"
Sep 11 14:49:59.352245 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:49:59.352215    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-rwvw4"
Sep 11 14:50:25.888440 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:25.888402    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"]
Sep 11 14:50:25.891530 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:25.891514    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:25.895169 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:25.895142    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cp687\""
Sep 11 14:50:25.895292 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:25.895196    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 14:50:25.895292 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:25.895200    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 11 14:50:25.895292 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:25.895236    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 14:50:25.902548 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:25.902523    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"]
Sep 11 14:50:25.931739 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:25.931694    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbfe44ff-9671-4f0d-a263-d9199c35c760-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5\" (UID: \"dbfe44ff-9671-4f0d-a263-d9199c35c760\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:25.931935 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:25.931763    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh4wk\" (UniqueName: \"kubernetes.io/projected/dbfe44ff-9671-4f0d-a263-d9199c35c760-kube-api-access-qh4wk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5\" (UID: \"dbfe44ff-9671-4f0d-a263-d9199c35c760\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:25.931935 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:25.931854    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbfe44ff-9671-4f0d-a263-d9199c35c760-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5\" (UID: \"dbfe44ff-9671-4f0d-a263-d9199c35c760\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:25.931935 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:25.931901    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbfe44ff-9671-4f0d-a263-d9199c35c760-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5\" (UID: \"dbfe44ff-9671-4f0d-a263-d9199c35c760\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:25.932062 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:25.931957    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbfe44ff-9671-4f0d-a263-d9199c35c760-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5\" (UID: \"dbfe44ff-9671-4f0d-a263-d9199c35c760\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:25.932062 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:25.932024    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbfe44ff-9671-4f0d-a263-d9199c35c760-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5\" (UID: \"dbfe44ff-9671-4f0d-a263-d9199c35c760\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:26.033031 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:26.032994    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbfe44ff-9671-4f0d-a263-d9199c35c760-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5\" (UID: \"dbfe44ff-9671-4f0d-a263-d9199c35c760\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:26.033031 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:26.033034    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbfe44ff-9671-4f0d-a263-d9199c35c760-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5\" (UID: \"dbfe44ff-9671-4f0d-a263-d9199c35c760\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:26.033283 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:26.033053    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh4wk\" (UniqueName: \"kubernetes.io/projected/dbfe44ff-9671-4f0d-a263-d9199c35c760-kube-api-access-qh4wk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5\" (UID: \"dbfe44ff-9671-4f0d-a263-d9199c35c760\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:26.033283 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:26.033095    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbfe44ff-9671-4f0d-a263-d9199c35c760-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5\" (UID: \"dbfe44ff-9671-4f0d-a263-d9199c35c760\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:26.033283 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:26.033124    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbfe44ff-9671-4f0d-a263-d9199c35c760-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5\" (UID: \"dbfe44ff-9671-4f0d-a263-d9199c35c760\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:26.033283 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:26.033153    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbfe44ff-9671-4f0d-a263-d9199c35c760-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5\" (UID: \"dbfe44ff-9671-4f0d-a263-d9199c35c760\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:26.033615 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:26.033580    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbfe44ff-9671-4f0d-a263-d9199c35c760-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5\" (UID: \"dbfe44ff-9671-4f0d-a263-d9199c35c760\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:26.033615 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:26.033598    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbfe44ff-9671-4f0d-a263-d9199c35c760-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5\" (UID: \"dbfe44ff-9671-4f0d-a263-d9199c35c760\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:26.033766 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:26.033581    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbfe44ff-9671-4f0d-a263-d9199c35c760-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5\" (UID: \"dbfe44ff-9671-4f0d-a263-d9199c35c760\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:26.035351 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:26.035325    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbfe44ff-9671-4f0d-a263-d9199c35c760-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5\" (UID: \"dbfe44ff-9671-4f0d-a263-d9199c35c760\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:26.035630 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:26.035608    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbfe44ff-9671-4f0d-a263-d9199c35c760-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5\" (UID: \"dbfe44ff-9671-4f0d-a263-d9199c35c760\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:26.042828 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:26.042796    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh4wk\" (UniqueName: \"kubernetes.io/projected/dbfe44ff-9671-4f0d-a263-d9199c35c760-kube-api-access-qh4wk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5\" (UID: \"dbfe44ff-9671-4f0d-a263-d9199c35c760\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:26.202179 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:26.202098    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:26.478378 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:26.478350    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"]
Sep 11 14:50:26.480866 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:50:26.480804    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbfe44ff_9671_4f0d_a263_d9199c35c760.slice/crio-47f865fb43e3559d1067b1d3a3cf3b8fd3cfc8765b7cec47bef2808d8e988e17 WatchSource:0}: Error finding container 47f865fb43e3559d1067b1d3a3cf3b8fd3cfc8765b7cec47bef2808d8e988e17: Status 404 returned error can't find the container with id 47f865fb43e3559d1067b1d3a3cf3b8fd3cfc8765b7cec47bef2808d8e988e17
Sep 11 14:50:26.482314 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:26.482291    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:50:26.523694 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:26.523659    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5" event={"ID":"dbfe44ff-9671-4f0d-a263-d9199c35c760","Type":"ContainerStarted","Data":"47f865fb43e3559d1067b1d3a3cf3b8fd3cfc8765b7cec47bef2808d8e988e17"}
Sep 11 14:50:28.538408 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:28.538370    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5" event={"ID":"dbfe44ff-9671-4f0d-a263-d9199c35c760","Type":"ContainerStarted","Data":"ae4feacc4cb2cf7e3393b8f7ad0e2e9c4e2a46ea25b1f1640c6be754028ebfe1"}
Sep 11 14:50:28.558151 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:28.558102    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5" podStartSLOduration=2.377056992 podStartE2EDuration="3.558087552s" totalImagesPullingTime="1.18103056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:50:28.55744748 +0000 UTC m=+608.345711820" watchObservedRunningTime="2026-09-11 14:50:28.558087552 +0000 UTC m=+608.346351850"
Sep 11 14:50:36.202921 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:36.202885    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:36.203396 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:36.202947    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:36.215392 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:36.215362    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:50:36.586006 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:50:36.585976    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5"
Sep 11 14:53:14.973845 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:14.973798    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"]
Sep 11 14:53:14.977654 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:14.977631    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:14.980426 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:14.980405    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-362f7a-openshift-default-dockercfg-h85vc\""
Sep 11 14:53:14.994759 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:14.994734    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"]
Sep 11 14:53:15.040760 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.040724    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-data\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.040983 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.040780    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-envoy\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.040983 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.040858    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-token\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.040983 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.040891    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-credential-socket\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.040983 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.040933    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-workload-certs\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.040983 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.040973    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-podinfo\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.041245 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.041008    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqnrn\" (UniqueName: \"kubernetes.io/projected/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-kube-api-access-hqnrn\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.041245 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.041101    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istiod-ca-cert\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.041245 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.041143    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-workload-socket\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.142449 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.142410    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-data\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.142623 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.142460    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-envoy\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.142623 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.142486    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-token\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.142623 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.142513    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-credential-socket\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.142623 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.142543    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-workload-certs\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.142623 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.142581    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-podinfo\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.142623 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.142612    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqnrn\" (UniqueName: \"kubernetes.io/projected/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-kube-api-access-hqnrn\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.142968 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.142679    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istiod-ca-cert\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.142968 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.142729    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-workload-socket\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.142968 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.142898    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-data\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.143116 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.143057    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-workload-certs\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.143235 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.143213    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-workload-socket\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.143379 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.143360    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-credential-socket\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.143554 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.143531    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istiod-ca-cert\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.145023 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.144996    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-envoy\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.145250 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.145234    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-podinfo\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.154675 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.154616    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-token\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.155119 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.155058    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqnrn\" (UniqueName: \"kubernetes.io/projected/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-kube-api-access-hqnrn\") pod \"e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") " pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.288207 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.288174    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:15.579792 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:15.579763    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"]
Sep 11 14:53:15.581703 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:53:15.581678    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f6cfcf_64db_4dfb_ae3b_bc8bc25fbf5c.slice/crio-c831ce82ca942ea7b03844f17ba5e6468abd718ab1054a2f1cfb9c42d56e5c4b WatchSource:0}: Error finding container c831ce82ca942ea7b03844f17ba5e6468abd718ab1054a2f1cfb9c42d56e5c4b: Status 404 returned error can't find the container with id c831ce82ca942ea7b03844f17ba5e6468abd718ab1054a2f1cfb9c42d56e5c4b
Sep 11 14:53:16.327423 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:16.327382    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4" event={"ID":"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c","Type":"ContainerStarted","Data":"c831ce82ca942ea7b03844f17ba5e6468abd718ab1054a2f1cfb9c42d56e5c4b"}
Sep 11 14:53:18.609680 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:18.609638    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:53:18.609945 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:18.609722    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:53:18.609945 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:18.609751    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:53:19.347975 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:19.347935    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4" event={"ID":"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c","Type":"ContainerStarted","Data":"d70a083c6ac54eca32a2ce9d317797669addbe3de1094d40fb2d46fc12e5a81d"}
Sep 11 14:53:19.371954 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:19.371900    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4" podStartSLOduration=2.346221349 podStartE2EDuration="5.371882948s" totalImagesPullingTime="3.025661599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:19.370680913 +0000 UTC m=+779.158945228" watchObservedRunningTime="2026-09-11 14:53:19.371882948 +0000 UTC m=+779.160147247"
Sep 11 14:53:19.390183 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:19.390146    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"]
Sep 11 14:53:19.393983 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:19.393958    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"
Sep 11 14:53:19.396777 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:19.396750    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-837632-serving-cert\""
Sep 11 14:53:19.397097 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:19.397081    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zzqqg\""
Sep 11 14:53:19.407012 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:19.406989    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"]
Sep 11 14:53:19.483307 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:19.483269    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd30c533-7634-4d21-a6e0-249dbba1e43c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4\" (UID: \"cd30c533-7634-4d21-a6e0-249dbba1e43c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"
Sep 11 14:53:19.483473 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:19.483320    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd30c533-7634-4d21-a6e0-249dbba1e43c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4\" (UID: \"cd30c533-7634-4d21-a6e0-249dbba1e43c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"
Sep 11 14:53:19.483473 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:19.483423    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd30c533-7634-4d21-a6e0-249dbba1e43c-metrics-certs\") pod \"maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4\" (UID: \"cd30c533-7634-4d21-a6e0-249dbba1e43c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"
Sep 11 14:53:19.483599 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:19.483471    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc22q\" (UniqueName: \"kubernetes.io/projected/cd30c533-7634-4d21-a6e0-249dbba1e43c-kube-api-access-lc22q\") pod \"maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4\" (UID: \"cd30c533-7634-4d21-a6e0-249dbba1e43c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"
Sep 11 14:53:19.584955 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:19.584918    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd30c533-7634-4d21-a6e0-249dbba1e43c-metrics-certs\") pod \"maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4\" (UID: \"cd30c533-7634-4d21-a6e0-249dbba1e43c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"
Sep 11 14:53:19.585139 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:19.584973    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc22q\" (UniqueName: \"kubernetes.io/projected/cd30c533-7634-4d21-a6e0-249dbba1e43c-kube-api-access-lc22q\") pod \"maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4\" (UID: \"cd30c533-7634-4d21-a6e0-249dbba1e43c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"
Sep 11 14:53:19.585139 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:19.585119    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd30c533-7634-4d21-a6e0-249dbba1e43c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4\" (UID: \"cd30c533-7634-4d21-a6e0-249dbba1e43c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"
Sep 11 14:53:19.585243 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:19.585167    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd30c533-7634-4d21-a6e0-249dbba1e43c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4\" (UID: \"cd30c533-7634-4d21-a6e0-249dbba1e43c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"
Sep 11 14:53:19.585314 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:53:19.585293    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-serving-cert: secret "maas-api-e2e-worker-w4-837632-serving-cert" not found
Sep 11 14:53:19.585382 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:53:19.585369    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd30c533-7634-4d21-a6e0-249dbba1e43c-maas-api-tls podName:cd30c533-7634-4d21-a6e0-249dbba1e43c nodeName:}" failed. No retries permitted until 2026-09-11 14:53:20.085348005 +0000 UTC m=+779.873612282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cd30c533-7634-4d21-a6e0-249dbba1e43c-maas-api-tls") pod "maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4" (UID: "cd30c533-7634-4d21-a6e0-249dbba1e43c") : secret "maas-api-e2e-worker-w4-837632-serving-cert" not found
Sep 11 14:53:19.585910 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:19.585885    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd30c533-7634-4d21-a6e0-249dbba1e43c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4\" (UID: \"cd30c533-7634-4d21-a6e0-249dbba1e43c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"
Sep 11 14:53:19.587636 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:19.587596    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd30c533-7634-4d21-a6e0-249dbba1e43c-metrics-certs\") pod \"maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4\" (UID: \"cd30c533-7634-4d21-a6e0-249dbba1e43c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"
Sep 11 14:53:19.593400 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:19.593373    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc22q\" (UniqueName: \"kubernetes.io/projected/cd30c533-7634-4d21-a6e0-249dbba1e43c-kube-api-access-lc22q\") pod \"maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4\" (UID: \"cd30c533-7634-4d21-a6e0-249dbba1e43c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"
Sep 11 14:53:20.091535 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.091503    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd30c533-7634-4d21-a6e0-249dbba1e43c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4\" (UID: \"cd30c533-7634-4d21-a6e0-249dbba1e43c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"
Sep 11 14:53:20.094476 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.094451    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd30c533-7634-4d21-a6e0-249dbba1e43c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4\" (UID: \"cd30c533-7634-4d21-a6e0-249dbba1e43c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"
Sep 11 14:53:20.207410 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.207375    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n"]
Sep 11 14:53:20.211193 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.211174    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n"
Sep 11 14:53:20.213719 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.213695    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-837632-dockercfg-d9pfw\""
Sep 11 14:53:20.213845 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.213697    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-837632\""
Sep 11 14:53:20.223599 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.223571    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n"]
Sep 11 14:53:20.235035 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.235007    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4"]
Sep 11 14:53:20.238506 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.238486    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4"
Sep 11 14:53:20.251890 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.251863    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4"]
Sep 11 14:53:20.288841 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.288800    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:20.293670 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.293635    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e3498c-b895-4e07-82af-9586e0ef670d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n\" (UID: \"74e3498c-b895-4e07-82af-9586e0ef670d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n"
Sep 11 14:53:20.293804 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.293683    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:20.293804 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.293688    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdj2b\" (UniqueName: \"kubernetes.io/projected/dead80a8-6b34-4408-ac1b-867c2ec0b91f-kube-api-access-xdj2b\") pod \"payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4\" (UID: \"dead80a8-6b34-4408-ac1b-867c2ec0b91f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4"
Sep 11 14:53:20.293804 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.293744    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqg2p\" (UniqueName: \"kubernetes.io/projected/74e3498c-b895-4e07-82af-9586e0ef670d-kube-api-access-dqg2p\") pod \"payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n\" (UID: \"74e3498c-b895-4e07-82af-9586e0ef670d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n"
Sep 11 14:53:20.293967 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.293884    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dead80a8-6b34-4408-ac1b-867c2ec0b91f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4\" (UID: \"dead80a8-6b34-4408-ac1b-867c2ec0b91f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4"
Sep 11 14:53:20.309281 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.309252    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"
Sep 11 14:53:20.353886 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.353792    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:20.354769 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.354745    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:20.395681 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.394870    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e3498c-b895-4e07-82af-9586e0ef670d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n\" (UID: \"74e3498c-b895-4e07-82af-9586e0ef670d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n"
Sep 11 14:53:20.395681 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.395490    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdj2b\" (UniqueName: \"kubernetes.io/projected/dead80a8-6b34-4408-ac1b-867c2ec0b91f-kube-api-access-xdj2b\") pod \"payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4\" (UID: \"dead80a8-6b34-4408-ac1b-867c2ec0b91f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4"
Sep 11 14:53:20.395681 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.395539    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqg2p\" (UniqueName: \"kubernetes.io/projected/74e3498c-b895-4e07-82af-9586e0ef670d-kube-api-access-dqg2p\") pod \"payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n\" (UID: \"74e3498c-b895-4e07-82af-9586e0ef670d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n"
Sep 11 14:53:20.396073 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.395699    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dead80a8-6b34-4408-ac1b-867c2ec0b91f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4\" (UID: \"dead80a8-6b34-4408-ac1b-867c2ec0b91f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4"
Sep 11 14:53:20.396855 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.396696    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dead80a8-6b34-4408-ac1b-867c2ec0b91f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4\" (UID: \"dead80a8-6b34-4408-ac1b-867c2ec0b91f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4"
Sep 11 14:53:20.397740 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.397696    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e3498c-b895-4e07-82af-9586e0ef670d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n\" (UID: \"74e3498c-b895-4e07-82af-9586e0ef670d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n"
Sep 11 14:53:20.405411 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.405384    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdj2b\" (UniqueName: \"kubernetes.io/projected/dead80a8-6b34-4408-ac1b-867c2ec0b91f-kube-api-access-xdj2b\") pod \"payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4\" (UID: \"dead80a8-6b34-4408-ac1b-867c2ec0b91f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4"
Sep 11 14:53:20.405540 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.405514    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqg2p\" (UniqueName: \"kubernetes.io/projected/74e3498c-b895-4e07-82af-9586e0ef670d-kube-api-access-dqg2p\") pod \"payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n\" (UID: \"74e3498c-b895-4e07-82af-9586e0ef670d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n"
Sep 11 14:53:20.525690 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.525662    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-837632-dockercfg-d9pfw\""
Sep 11 14:53:20.533140 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.533111    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n"
Sep 11 14:53:20.548707 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.548680    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4"
Sep 11 14:53:20.595664 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.594304    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"]
Sep 11 14:53:20.601140 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:53:20.601102    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd30c533_7634_4d21_a6e0_249dbba1e43c.slice/crio-3481019ee1d6426129c570de979fa3d5cf3c743f49f583c2ea35c21960eed6b3 WatchSource:0}: Error finding container 3481019ee1d6426129c570de979fa3d5cf3c743f49f583c2ea35c21960eed6b3: Status 404 returned error can't find the container with id 3481019ee1d6426129c570de979fa3d5cf3c743f49f583c2ea35c21960eed6b3
Sep 11 14:53:20.820264 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.820235    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n"]
Sep 11 14:53:20.821672 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:53:20.821643    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e3498c_b895_4e07_82af_9586e0ef670d.slice/crio-ab4d73604d67616e478dfefa8c147cc7e12647f5d8249f3cece0bf09844a3308 WatchSource:0}: Error finding container ab4d73604d67616e478dfefa8c147cc7e12647f5d8249f3cece0bf09844a3308: Status 404 returned error can't find the container with id ab4d73604d67616e478dfefa8c147cc7e12647f5d8249f3cece0bf09844a3308
Sep 11 14:53:20.841444 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.841409    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4"]
Sep 11 14:53:20.843485 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:53:20.843212    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddead80a8_6b34_4408_ac1b_867c2ec0b91f.slice/crio-2b40f0b069b6d31cf21b5564adf8d2f1d68e78965575298760015d09e9882740 WatchSource:0}: Error finding container 2b40f0b069b6d31cf21b5564adf8d2f1d68e78965575298760015d09e9882740: Status 404 returned error can't find the container with id 2b40f0b069b6d31cf21b5564adf8d2f1d68e78965575298760015d09e9882740
Sep 11 14:53:20.886798 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.886770    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"]
Sep 11 14:53:20.892306 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.892285    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"
Sep 11 14:53:20.894999 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.894977    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-e7ca56-serving-cert\""
Sep 11 14:53:20.911225 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:20.911060    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"]
Sep 11 14:53:21.005716 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.005679    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbc975c-be6f-4326-b8a1-b2de058f4086-metrics-certs\") pod \"maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf\" (UID: \"0cbc975c-be6f-4326-b8a1-b2de058f4086\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"
Sep 11 14:53:21.005915 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.005720    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbsvf\" (UniqueName: \"kubernetes.io/projected/0cbc975c-be6f-4326-b8a1-b2de058f4086-kube-api-access-mbsvf\") pod \"maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf\" (UID: \"0cbc975c-be6f-4326-b8a1-b2de058f4086\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"
Sep 11 14:53:21.005915 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.005758    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cbc975c-be6f-4326-b8a1-b2de058f4086-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf\" (UID: \"0cbc975c-be6f-4326-b8a1-b2de058f4086\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"
Sep 11 14:53:21.005915 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.005895    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbc975c-be6f-4326-b8a1-b2de058f4086-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf\" (UID: \"0cbc975c-be6f-4326-b8a1-b2de058f4086\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"
Sep 11 14:53:21.106576 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.106540    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbc975c-be6f-4326-b8a1-b2de058f4086-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf\" (UID: \"0cbc975c-be6f-4326-b8a1-b2de058f4086\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"
Sep 11 14:53:21.107007 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.106653    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbc975c-be6f-4326-b8a1-b2de058f4086-metrics-certs\") pod \"maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf\" (UID: \"0cbc975c-be6f-4326-b8a1-b2de058f4086\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"
Sep 11 14:53:21.107007 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.106681    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbsvf\" (UniqueName: \"kubernetes.io/projected/0cbc975c-be6f-4326-b8a1-b2de058f4086-kube-api-access-mbsvf\") pod \"maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf\" (UID: \"0cbc975c-be6f-4326-b8a1-b2de058f4086\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"
Sep 11 14:53:21.107007 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.106718    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cbc975c-be6f-4326-b8a1-b2de058f4086-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf\" (UID: \"0cbc975c-be6f-4326-b8a1-b2de058f4086\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"
Sep 11 14:53:21.107007 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:53:21.106846    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-serving-cert: secret "maas-api-e2e-worker-w0-e7ca56-serving-cert" not found
Sep 11 14:53:21.107007 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:53:21.106910    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cbc975c-be6f-4326-b8a1-b2de058f4086-maas-api-tls podName:0cbc975c-be6f-4326-b8a1-b2de058f4086 nodeName:}" failed. No retries permitted until 2026-09-11 14:53:21.606889982 +0000 UTC m=+781.395154272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0cbc975c-be6f-4326-b8a1-b2de058f4086-maas-api-tls") pod "maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf" (UID: "0cbc975c-be6f-4326-b8a1-b2de058f4086") : secret "maas-api-e2e-worker-w0-e7ca56-serving-cert" not found
Sep 11 14:53:21.107364 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.107337    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbc975c-be6f-4326-b8a1-b2de058f4086-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf\" (UID: \"0cbc975c-be6f-4326-b8a1-b2de058f4086\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"
Sep 11 14:53:21.109600 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.109552    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbc975c-be6f-4326-b8a1-b2de058f4086-metrics-certs\") pod \"maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf\" (UID: \"0cbc975c-be6f-4326-b8a1-b2de058f4086\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"
Sep 11 14:53:21.121100 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.121009    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbsvf\" (UniqueName: \"kubernetes.io/projected/0cbc975c-be6f-4326-b8a1-b2de058f4086-kube-api-access-mbsvf\") pod \"maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf\" (UID: \"0cbc975c-be6f-4326-b8a1-b2de058f4086\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"
Sep 11 14:53:21.361118 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.361026    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4" event={"ID":"dead80a8-6b34-4408-ac1b-867c2ec0b91f","Type":"ContainerStarted","Data":"6c8965089c4cc0a1475c2cd7159c6a70596bef7a9e0ea7e1f9de594e22f283f1"}
Sep 11 14:53:21.361118 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.361087    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4" event={"ID":"dead80a8-6b34-4408-ac1b-867c2ec0b91f","Type":"ContainerStarted","Data":"2b40f0b069b6d31cf21b5564adf8d2f1d68e78965575298760015d09e9882740"}
Sep 11 14:53:21.361338 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.361128    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4"
Sep 11 14:53:21.362762 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.362733    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n" event={"ID":"74e3498c-b895-4e07-82af-9586e0ef670d","Type":"ContainerStarted","Data":"1c86b7bea5734958e8abeed7cf63dba00d37e4a2ad03cbf3b35790ecfc91254c"}
Sep 11 14:53:21.362894 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.362779    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n" event={"ID":"74e3498c-b895-4e07-82af-9586e0ef670d","Type":"ContainerStarted","Data":"ab4d73604d67616e478dfefa8c147cc7e12647f5d8249f3cece0bf09844a3308"}
Sep 11 14:53:21.362894 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.362866    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n"
Sep 11 14:53:21.364303 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.364266    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4" event={"ID":"cd30c533-7634-4d21-a6e0-249dbba1e43c","Type":"ContainerStarted","Data":"7c3c064ed651dbb53c815b389537c9ababf72d204f43898f45088a07673db73c"}
Sep 11 14:53:21.364415 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.364312    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4" event={"ID":"cd30c533-7634-4d21-a6e0-249dbba1e43c","Type":"ContainerStarted","Data":"3481019ee1d6426129c570de979fa3d5cf3c743f49f583c2ea35c21960eed6b3"}
Sep 11 14:53:21.364475 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.364439    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"
Sep 11 14:53:21.385985 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.385915    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4" podStartSLOduration=1.38590166 podStartE2EDuration="1.38590166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:21.384191215 +0000 UTC m=+781.172455514" watchObservedRunningTime="2026-09-11 14:53:21.38590166 +0000 UTC m=+781.174165967"
Sep 11 14:53:21.411111 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.411061    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4" podStartSLOduration=1.875573916 podStartE2EDuration="2.411044815s" totalImagesPullingTime="535.470899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:21.410725359 +0000 UTC m=+781.198989657" watchObservedRunningTime="2026-09-11 14:53:21.411044815 +0000 UTC m=+781.199309158"
Sep 11 14:53:21.434575 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.434507    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n" podStartSLOduration=1.434486708 podStartE2EDuration="1.434486708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:21.432728055 +0000 UTC m=+781.220992356" watchObservedRunningTime="2026-09-11 14:53:21.434486708 +0000 UTC m=+781.222751009"
Sep 11 14:53:21.611390 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.611307    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cbc975c-be6f-4326-b8a1-b2de058f4086-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf\" (UID: \"0cbc975c-be6f-4326-b8a1-b2de058f4086\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"
Sep 11 14:53:21.614300 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.614273    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cbc975c-be6f-4326-b8a1-b2de058f4086-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf\" (UID: \"0cbc975c-be6f-4326-b8a1-b2de058f4086\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"
Sep 11 14:53:21.818543 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.818504    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"
Sep 11 14:53:21.907580 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.907543    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv"]
Sep 11 14:53:21.912495 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.912464    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv"
Sep 11 14:53:21.915357 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.915332    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-e7ca56\""
Sep 11 14:53:21.915485 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.915380    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-e7ca56-dockercfg-54ns2\""
Sep 11 14:53:21.924072 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:21.924042    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv"]
Sep 11 14:53:22.016582 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:22.016544    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80136c17-90ca-4571-9164-f25620470160-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv\" (UID: \"80136c17-90ca-4571-9164-f25620470160\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv"
Sep 11 14:53:22.016754 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:22.016591    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzlcz\" (UniqueName: \"kubernetes.io/projected/80136c17-90ca-4571-9164-f25620470160-kube-api-access-bzlcz\") pod \"payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv\" (UID: \"80136c17-90ca-4571-9164-f25620470160\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv"
Sep 11 14:53:22.098424 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:22.098396    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"]
Sep 11 14:53:22.100728 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:53:22.100700    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbc975c_be6f_4326_b8a1_b2de058f4086.slice/crio-9af94bff9f260db806df606dc9a030d17b9b784e21e47058f3fa7f36bc869f11 WatchSource:0}: Error finding container 9af94bff9f260db806df606dc9a030d17b9b784e21e47058f3fa7f36bc869f11: Status 404 returned error can't find the container with id 9af94bff9f260db806df606dc9a030d17b9b784e21e47058f3fa7f36bc869f11
Sep 11 14:53:22.118169 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:22.118135    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80136c17-90ca-4571-9164-f25620470160-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv\" (UID: \"80136c17-90ca-4571-9164-f25620470160\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv"
Sep 11 14:53:22.118556 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:22.118183    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzlcz\" (UniqueName: \"kubernetes.io/projected/80136c17-90ca-4571-9164-f25620470160-kube-api-access-bzlcz\") pod \"payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv\" (UID: \"80136c17-90ca-4571-9164-f25620470160\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv"
Sep 11 14:53:22.119031 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:22.119009    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80136c17-90ca-4571-9164-f25620470160-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv\" (UID: \"80136c17-90ca-4571-9164-f25620470160\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv"
Sep 11 14:53:22.127784 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:22.127724    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzlcz\" (UniqueName: \"kubernetes.io/projected/80136c17-90ca-4571-9164-f25620470160-kube-api-access-bzlcz\") pod \"payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv\" (UID: \"80136c17-90ca-4571-9164-f25620470160\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv"
Sep 11 14:53:22.225677 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:22.225637    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv"
Sep 11 14:53:22.371487 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:22.371445    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf" event={"ID":"0cbc975c-be6f-4326-b8a1-b2de058f4086","Type":"ContainerStarted","Data":"9af94bff9f260db806df606dc9a030d17b9b784e21e47058f3fa7f36bc869f11"}
Sep 11 14:53:22.502562 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:22.502531    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv"]
Sep 11 14:53:22.504584 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:53:22.504559    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80136c17_90ca_4571_9164_f25620470160.slice/crio-7a986c60d64af922b3206cc59f566866d0f31ff111162914e62737da27db115a WatchSource:0}: Error finding container 7a986c60d64af922b3206cc59f566866d0f31ff111162914e62737da27db115a: Status 404 returned error can't find the container with id 7a986c60d64af922b3206cc59f566866d0f31ff111162914e62737da27db115a
Sep 11 14:53:23.378528 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:23.378492    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv" event={"ID":"80136c17-90ca-4571-9164-f25620470160","Type":"ContainerStarted","Data":"4f0945ce85084fa08c576dde6f3952eaec5e417740775d14b8830318864c7305"}
Sep 11 14:53:23.379028 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:23.378548    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv" event={"ID":"80136c17-90ca-4571-9164-f25620470160","Type":"ContainerStarted","Data":"7a986c60d64af922b3206cc59f566866d0f31ff111162914e62737da27db115a"}
Sep 11 14:53:23.379028 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:23.378672    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv"
Sep 11 14:53:23.380347 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:23.380316    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf" event={"ID":"0cbc975c-be6f-4326-b8a1-b2de058f4086","Type":"ContainerStarted","Data":"97a558d8edf0caa73c6220a5bcdcc685ebbe0256f158e160a0112e2fbba274b7"}
Sep 11 14:53:23.380452 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:23.380396    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"
Sep 11 14:53:23.408993 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:23.408930    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv" podStartSLOduration=2.408909538 podStartE2EDuration="2.408909538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:23.401395678 +0000 UTC m=+783.189660003" watchObservedRunningTime="2026-09-11 14:53:23.408909538 +0000 UTC m=+783.197173838"
Sep 11 14:53:23.432850 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:23.432727    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf" podStartSLOduration=3.143522707 podStartE2EDuration="3.432693498s" totalImagesPullingTime="289.170791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:23.43013955 +0000 UTC m=+783.218403858" watchObservedRunningTime="2026-09-11 14:53:23.432693498 +0000 UTC m=+783.220957787"
Sep 11 14:53:25.305504 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:25.305469    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"]
Sep 11 14:53:25.310172 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:25.310142    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"
Sep 11 14:53:25.314041 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:25.314013    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-362f7a-serving-cert\""
Sep 11 14:53:25.321806 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:25.321754    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"]
Sep 11 14:53:25.460190 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:25.460150    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-openshift-service-ca\") pod \"maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56\" (UID: \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"
Sep 11 14:53:25.460355 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:25.460219    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-metrics-certs\") pod \"maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56\" (UID: \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"
Sep 11 14:53:25.460355 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:25.460266    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp2r6\" (UniqueName: \"kubernetes.io/projected/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-kube-api-access-fp2r6\") pod \"maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56\" (UID: \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"
Sep 11 14:53:25.460355 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:25.460337    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-maas-api-tls\") pod \"maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56\" (UID: \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"
Sep 11 14:53:25.561795 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:25.561694    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-metrics-certs\") pod \"maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56\" (UID: \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"
Sep 11 14:53:25.561795 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:25.561737    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp2r6\" (UniqueName: \"kubernetes.io/projected/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-kube-api-access-fp2r6\") pod \"maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56\" (UID: \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"
Sep 11 14:53:25.561795 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:25.561780    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-maas-api-tls\") pod \"maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56\" (UID: \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"
Sep 11 14:53:25.562082 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:25.561893    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-openshift-service-ca\") pod \"maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56\" (UID: \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"
Sep 11 14:53:25.562082 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:53:25.561977    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-serving-cert: secret "maas-api-e2e-inf-a-362f7a-serving-cert" not found
Sep 11 14:53:25.562082 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:53:25.562058    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-maas-api-tls podName:2daa6ec8-8f7b-4532-a7a5-192cd9df0793 nodeName:}" failed. No retries permitted until 2026-09-11 14:53:26.062037542 +0000 UTC m=+785.850301829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-maas-api-tls") pod "maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56" (UID: "2daa6ec8-8f7b-4532-a7a5-192cd9df0793") : secret "maas-api-e2e-inf-a-362f7a-serving-cert" not found
Sep 11 14:53:25.562639 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:25.562614    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-openshift-service-ca\") pod \"maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56\" (UID: \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"
Sep 11 14:53:25.564343 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:25.564317    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-metrics-certs\") pod \"maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56\" (UID: \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"
Sep 11 14:53:25.573944 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:25.573921    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp2r6\" (UniqueName: \"kubernetes.io/projected/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-kube-api-access-fp2r6\") pod \"maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56\" (UID: \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"
Sep 11 14:53:26.066542 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:26.066504    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-maas-api-tls\") pod \"maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56\" (UID: \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"
Sep 11 14:53:26.069025 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:26.069002    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-maas-api-tls\") pod \"maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56\" (UID: \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"
Sep 11 14:53:26.228965 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:26.228930    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"
Sep 11 14:53:26.512842 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:26.512551    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"]
Sep 11 14:53:26.515653 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:53:26.515619    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2daa6ec8_8f7b_4532_a7a5_192cd9df0793.slice/crio-596e67e24628a72341984a5e5aa5aab469851ed8e8c30b2e385486fdcb6345ec WatchSource:0}: Error finding container 596e67e24628a72341984a5e5aa5aab469851ed8e8c30b2e385486fdcb6345ec: Status 404 returned error can't find the container with id 596e67e24628a72341984a5e5aa5aab469851ed8e8c30b2e385486fdcb6345ec
Sep 11 14:53:27.377685 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:27.377655    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"
Sep 11 14:53:27.407088 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:27.407032    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56" event={"ID":"2daa6ec8-8f7b-4532-a7a5-192cd9df0793","Type":"ContainerStarted","Data":"bb5ff971effee8ad3a3db06956bf92e50299e7fedb408f624d5e7c4156fc72a0"}
Sep 11 14:53:27.407269 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:27.407108    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"
Sep 11 14:53:27.407269 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:27.407140    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56" event={"ID":"2daa6ec8-8f7b-4532-a7a5-192cd9df0793","Type":"ContainerStarted","Data":"596e67e24628a72341984a5e5aa5aab469851ed8e8c30b2e385486fdcb6345ec"}
Sep 11 14:53:27.442938 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:27.442874    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56" podStartSLOduration=2.124575411 podStartE2EDuration="2.4428521s" totalImagesPullingTime="318.276689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:27.439199744 +0000 UTC m=+787.227464042" watchObservedRunningTime="2026-09-11 14:53:27.4428521 +0000 UTC m=+787.231116399"
Sep 11 14:53:29.391972 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:29.391936    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"
Sep 11 14:53:31.836108 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:31.836065    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx"]
Sep 11 14:53:31.840124 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:31.840060    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx"
Sep 11 14:53:31.842856 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:31.842830    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-3219e3-dockercfg-5c55m\""
Sep 11 14:53:31.843012 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:31.842810    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-3219e3\""
Sep 11 14:53:31.851979 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:31.851953    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx"]
Sep 11 14:53:31.928268 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:31.928222    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22234bfd-e677-4897-b5a4-2194b9fe73dc-plugins-config-volume\") pod \"payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx\" (UID: \"22234bfd-e677-4897-b5a4-2194b9fe73dc\") " pod="openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx"
Sep 11 14:53:31.928453 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:31.928290    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2k9h\" (UniqueName: \"kubernetes.io/projected/22234bfd-e677-4897-b5a4-2194b9fe73dc-kube-api-access-n2k9h\") pod \"payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx\" (UID: \"22234bfd-e677-4897-b5a4-2194b9fe73dc\") " pod="openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx"
Sep 11 14:53:32.029374 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:32.029335    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22234bfd-e677-4897-b5a4-2194b9fe73dc-plugins-config-volume\") pod \"payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx\" (UID: \"22234bfd-e677-4897-b5a4-2194b9fe73dc\") " pod="openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx"
Sep 11 14:53:32.029537 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:32.029386    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2k9h\" (UniqueName: \"kubernetes.io/projected/22234bfd-e677-4897-b5a4-2194b9fe73dc-kube-api-access-n2k9h\") pod \"payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx\" (UID: \"22234bfd-e677-4897-b5a4-2194b9fe73dc\") " pod="openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx"
Sep 11 14:53:32.030143 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:32.030121    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22234bfd-e677-4897-b5a4-2194b9fe73dc-plugins-config-volume\") pod \"payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx\" (UID: \"22234bfd-e677-4897-b5a4-2194b9fe73dc\") " pod="openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx"
Sep 11 14:53:32.038598 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:32.038573    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2k9h\" (UniqueName: \"kubernetes.io/projected/22234bfd-e677-4897-b5a4-2194b9fe73dc-kube-api-access-n2k9h\") pod \"payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx\" (UID: \"22234bfd-e677-4897-b5a4-2194b9fe73dc\") " pod="openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx"
Sep 11 14:53:32.151574 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:32.151483    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx"
Sep 11 14:53:32.373730 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:32.373701    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n"
Sep 11 14:53:32.373917 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:32.373776    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4"
Sep 11 14:53:32.435097 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:32.434962    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx"]
Sep 11 14:53:32.444497 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:53:32.444456    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22234bfd_e677_4897_b5a4_2194b9fe73dc.slice/crio-59366daa914a0316eb1813fc8ceb551cb7a02074774bc49902ac39c001a4ec2e WatchSource:0}: Error finding container 59366daa914a0316eb1813fc8ceb551cb7a02074774bc49902ac39c001a4ec2e: Status 404 returned error can't find the container with id 59366daa914a0316eb1813fc8ceb551cb7a02074774bc49902ac39c001a4ec2e
Sep 11 14:53:33.418651 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:33.418615    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"
Sep 11 14:53:33.453719 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:33.453683    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx" event={"ID":"22234bfd-e677-4897-b5a4-2194b9fe73dc","Type":"ContainerStarted","Data":"6f752f7f978d6d1a66efc301aee3d455ea2a672496f994468dc08a6a3742e387"}
Sep 11 14:53:33.453963 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:33.453749    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx" event={"ID":"22234bfd-e677-4897-b5a4-2194b9fe73dc","Type":"ContainerStarted","Data":"59366daa914a0316eb1813fc8ceb551cb7a02074774bc49902ac39c001a4ec2e"}
Sep 11 14:53:33.453963 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:33.453846    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx"
Sep 11 14:53:33.476898 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:33.476516    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx" podStartSLOduration=2.476495352 podStartE2EDuration="2.476495352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:33.476445162 +0000 UTC m=+793.264709462" watchObservedRunningTime="2026-09-11 14:53:33.476495352 +0000 UTC m=+793.264759652"
Sep 11 14:53:34.388475 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:34.388440    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv"
Sep 11 14:53:34.781095 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:34.781061    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v"]
Sep 11 14:53:34.785212 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:34.785153    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v"
Sep 11 14:53:34.788178 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:34.788156    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f722bf59-dockercfg-65tmg\""
Sep 11 14:53:34.788306 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:34.788176    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f722bf59\""
Sep 11 14:53:34.795266 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:34.795245    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v"]
Sep 11 14:53:34.956057 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:34.956019    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnfw4\" (UniqueName: \"kubernetes.io/projected/24cf5ebe-57d5-41fc-a393-37d0a0bbfed5-kube-api-access-qnfw4\") pod \"payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v\" (UID: \"24cf5ebe-57d5-41fc-a393-37d0a0bbfed5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v"
Sep 11 14:53:34.956244 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:34.956208    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24cf5ebe-57d5-41fc-a393-37d0a0bbfed5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v\" (UID: \"24cf5ebe-57d5-41fc-a393-37d0a0bbfed5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v"
Sep 11 14:53:35.057517 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:35.057422    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnfw4\" (UniqueName: \"kubernetes.io/projected/24cf5ebe-57d5-41fc-a393-37d0a0bbfed5-kube-api-access-qnfw4\") pod \"payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v\" (UID: \"24cf5ebe-57d5-41fc-a393-37d0a0bbfed5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v"
Sep 11 14:53:35.057705 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:35.057530    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24cf5ebe-57d5-41fc-a393-37d0a0bbfed5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v\" (UID: \"24cf5ebe-57d5-41fc-a393-37d0a0bbfed5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v"
Sep 11 14:53:35.058198 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:35.058179    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24cf5ebe-57d5-41fc-a393-37d0a0bbfed5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v\" (UID: \"24cf5ebe-57d5-41fc-a393-37d0a0bbfed5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v"
Sep 11 14:53:35.066603 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:35.066577    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnfw4\" (UniqueName: \"kubernetes.io/projected/24cf5ebe-57d5-41fc-a393-37d0a0bbfed5-kube-api-access-qnfw4\") pod \"payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v\" (UID: \"24cf5ebe-57d5-41fc-a393-37d0a0bbfed5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v"
Sep 11 14:53:35.096667 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:35.096629    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v"
Sep 11 14:53:35.403185 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:35.403158    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v"]
Sep 11 14:53:35.404738 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:53:35.404711    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24cf5ebe_57d5_41fc_a393_37d0a0bbfed5.slice/crio-c579f368baeb77acaca9a2ce93cdc133a00760b18efa70c61795134ce9927ee4 WatchSource:0}: Error finding container c579f368baeb77acaca9a2ce93cdc133a00760b18efa70c61795134ce9927ee4: Status 404 returned error can't find the container with id c579f368baeb77acaca9a2ce93cdc133a00760b18efa70c61795134ce9927ee4
Sep 11 14:53:35.464939 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:35.464914    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v" event={"ID":"24cf5ebe-57d5-41fc-a393-37d0a0bbfed5","Type":"ContainerStarted","Data":"c579f368baeb77acaca9a2ce93cdc133a00760b18efa70c61795134ce9927ee4"}
Sep 11 14:53:36.471582 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:36.471544    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v" event={"ID":"24cf5ebe-57d5-41fc-a393-37d0a0bbfed5","Type":"ContainerStarted","Data":"4b04bb24a5932bbe145c15aa017f7c6f343cfcd0722b3d9ac4c53de149cddff3"}
Sep 11 14:53:36.472063 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:36.471692    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v"
Sep 11 14:53:36.490693 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:36.490636    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v" podStartSLOduration=2.490620217 podStartE2EDuration="2.490620217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:36.490299065 +0000 UTC m=+796.278563365" watchObservedRunningTime="2026-09-11 14:53:36.490620217 +0000 UTC m=+796.278884516"
Sep 11 14:53:44.461479 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:44.461405    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx"
Sep 11 14:53:45.524115 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:45.524022    2568 generic.go:356] "Generic (PLEG): container finished" podID="52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" containerID="d70a083c6ac54eca32a2ce9d317797669addbe3de1094d40fb2d46fc12e5a81d" exitCode=0
Sep 11 14:53:45.524485 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:45.524102    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4" event={"ID":"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c","Type":"ContainerDied","Data":"d70a083c6ac54eca32a2ce9d317797669addbe3de1094d40fb2d46fc12e5a81d"}
Sep 11 14:53:45.524673 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:45.524655    2568 scope.go:117] "RemoveContainer" containerID="d70a083c6ac54eca32a2ce9d317797669addbe3de1094d40fb2d46fc12e5a81d"
Sep 11 14:53:45.525918 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:45.525888    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:53:45.525982 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:45.525959    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:53:45.526020 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:45.525990    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:53:46.534806 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:46.534759    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4" event={"ID":"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c","Type":"ContainerStarted","Data":"af00ece293f05f0ade3121c71ad983f536bbbdffaa1d0bafab1ecc7d311b5685"}
Sep 11 14:53:47.288849 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:47.288797    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:47.289885 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:47.289850    2568 patch_prober.go:28] interesting pod/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 11 14:53:47.289984 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:47.289925    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4" podUID="52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 11 14:53:47.480652 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:47.480618    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v"
Sep 11 14:53:48.289048 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:48.289003    2568 patch_prober.go:28] interesting pod/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 11 14:53:48.289528 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:48.289113    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4" podUID="52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 11 14:53:49.289337 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:49.289300    2568 patch_prober.go:28] interesting pod/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 11 14:53:49.289701 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:49.289364    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4" podUID="52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 11 14:53:50.289276 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.289239    2568 patch_prober.go:28] interesting pod/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 11 14:53:50.289461 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.289295    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4" podUID="52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 11 14:53:50.353958 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.353917    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:50.459854 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.459796    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v"]
Sep 11 14:53:50.463632 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.463609    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v"
Sep 11 14:53:50.466623 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.466600    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-90d6f5\""
Sep 11 14:53:50.466736 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.466598    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-90d6f5-dockercfg-9mwrw\""
Sep 11 14:53:50.475345 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.475324    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v"]
Sep 11 14:53:50.494536 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.494506    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk"]
Sep 11 14:53:50.499735 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.499708    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk"
Sep 11 14:53:50.508969 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.508945    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk"]
Sep 11 14:53:50.545031 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.544924    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hdd2\" (UniqueName: \"kubernetes.io/projected/2341a23b-500e-4832-abf6-b3c858e84e0a-kube-api-access-2hdd2\") pod \"payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v\" (UID: \"2341a23b-500e-4832-abf6-b3c858e84e0a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v"
Sep 11 14:53:50.545201 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.545104    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2341a23b-500e-4832-abf6-b3c858e84e0a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v\" (UID: \"2341a23b-500e-4832-abf6-b3c858e84e0a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v"
Sep 11 14:53:50.646067 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.646026    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2b65\" (UniqueName: \"kubernetes.io/projected/c560d780-16cf-4739-851a-d034eeb090bb-kube-api-access-v2b65\") pod \"payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk\" (UID: \"c560d780-16cf-4739-851a-d034eeb090bb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk"
Sep 11 14:53:50.646241 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.646150    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hdd2\" (UniqueName: \"kubernetes.io/projected/2341a23b-500e-4832-abf6-b3c858e84e0a-kube-api-access-2hdd2\") pod \"payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v\" (UID: \"2341a23b-500e-4832-abf6-b3c858e84e0a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v"
Sep 11 14:53:50.646332 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.646308    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c560d780-16cf-4739-851a-d034eeb090bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk\" (UID: \"c560d780-16cf-4739-851a-d034eeb090bb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk"
Sep 11 14:53:50.646387 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.646356    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2341a23b-500e-4832-abf6-b3c858e84e0a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v\" (UID: \"2341a23b-500e-4832-abf6-b3c858e84e0a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v"
Sep 11 14:53:50.647645 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.647621    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2341a23b-500e-4832-abf6-b3c858e84e0a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v\" (UID: \"2341a23b-500e-4832-abf6-b3c858e84e0a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v"
Sep 11 14:53:50.656287 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.656254    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hdd2\" (UniqueName: \"kubernetes.io/projected/2341a23b-500e-4832-abf6-b3c858e84e0a-kube-api-access-2hdd2\") pod \"payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v\" (UID: \"2341a23b-500e-4832-abf6-b3c858e84e0a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v"
Sep 11 14:53:50.747702 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.747659    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c560d780-16cf-4739-851a-d034eeb090bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk\" (UID: \"c560d780-16cf-4739-851a-d034eeb090bb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk"
Sep 11 14:53:50.747892 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.747842    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2b65\" (UniqueName: \"kubernetes.io/projected/c560d780-16cf-4739-851a-d034eeb090bb-kube-api-access-v2b65\") pod \"payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk\" (UID: \"c560d780-16cf-4739-851a-d034eeb090bb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk"
Sep 11 14:53:50.748533 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.748505    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c560d780-16cf-4739-851a-d034eeb090bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk\" (UID: \"c560d780-16cf-4739-851a-d034eeb090bb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk"
Sep 11 14:53:50.764211 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.764179    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2b65\" (UniqueName: \"kubernetes.io/projected/c560d780-16cf-4739-851a-d034eeb090bb-kube-api-access-v2b65\") pod \"payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk\" (UID: \"c560d780-16cf-4739-851a-d034eeb090bb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk"
Sep 11 14:53:50.774578 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.774554    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v"
Sep 11 14:53:50.811661 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:50.811583    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk"
Sep 11 14:53:51.106221 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:51.106194    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v"]
Sep 11 14:53:51.108069 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:53:51.108032    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2341a23b_500e_4832_abf6_b3c858e84e0a.slice/crio-ef629f572752fc762dfff8663d0e0c9e639f6e751492c7c49d928a52e0410cf2 WatchSource:0}: Error finding container ef629f572752fc762dfff8663d0e0c9e639f6e751492c7c49d928a52e0410cf2: Status 404 returned error can't find the container with id ef629f572752fc762dfff8663d0e0c9e639f6e751492c7c49d928a52e0410cf2
Sep 11 14:53:51.167733 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:51.166040    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk"]
Sep 11 14:53:51.289025 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:51.288983    2568 patch_prober.go:28] interesting pod/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 11 14:53:51.289172 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:51.289082    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4" podUID="52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 11 14:53:51.570611 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:51.570570    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk" event={"ID":"c560d780-16cf-4739-851a-d034eeb090bb","Type":"ContainerStarted","Data":"e8a183489291889a9c43de7d9e8b3e61b253505703a744968eaeb07c50d71c5d"}
Sep 11 14:53:51.571111 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:51.570637    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk" event={"ID":"c560d780-16cf-4739-851a-d034eeb090bb","Type":"ContainerStarted","Data":"72cfbec9151fb61facd49963cd4457edad601132e7656d54cfe68754f456fa32"}
Sep 11 14:53:51.571111 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:51.570700    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk"
Sep 11 14:53:51.572359 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:51.572328    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v" event={"ID":"2341a23b-500e-4832-abf6-b3c858e84e0a","Type":"ContainerStarted","Data":"3fa7d21939e6d3f1e3f6bc899435edcfe2f3c70840bc3980e4a1061cb704fb7a"}
Sep 11 14:53:51.572472 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:51.572371    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v" event={"ID":"2341a23b-500e-4832-abf6-b3c858e84e0a","Type":"ContainerStarted","Data":"ef629f572752fc762dfff8663d0e0c9e639f6e751492c7c49d928a52e0410cf2"}
Sep 11 14:53:51.572771 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:51.572723    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v"
Sep 11 14:53:51.592607 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:51.592564    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk" podStartSLOduration=1.592552327 podStartE2EDuration="1.592552327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:51.590334081 +0000 UTC m=+811.378598380" watchObservedRunningTime="2026-09-11 14:53:51.592552327 +0000 UTC m=+811.380816622"
Sep 11 14:53:51.612256 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:51.612204    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v" podStartSLOduration=1.612188103 podStartE2EDuration="1.612188103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:51.611969589 +0000 UTC m=+811.400233909" watchObservedRunningTime="2026-09-11 14:53:51.612188103 +0000 UTC m=+811.400452418"
Sep 11 14:53:52.292902 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:52.292870    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:53:52.293890 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:53:52.293871    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:54:01.416236 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:01.416197    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v"]
Sep 11 14:54:01.416736 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:01.416439    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v" podUID="24cf5ebe-57d5-41fc-a393-37d0a0bbfed5" containerName="payload-pre-processing" containerID="cri-o://4b04bb24a5932bbe145c15aa017f7c6f343cfcd0722b3d9ac4c53de149cddff3" gracePeriod=30
Sep 11 14:54:01.631915 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:01.631867    2568 generic.go:356] "Generic (PLEG): container finished" podID="24cf5ebe-57d5-41fc-a393-37d0a0bbfed5" containerID="4b04bb24a5932bbe145c15aa017f7c6f343cfcd0722b3d9ac4c53de149cddff3" exitCode=0
Sep 11 14:54:01.632088 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:01.631918    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v" event={"ID":"24cf5ebe-57d5-41fc-a393-37d0a0bbfed5","Type":"ContainerDied","Data":"4b04bb24a5932bbe145c15aa017f7c6f343cfcd0722b3d9ac4c53de149cddff3"}
Sep 11 14:54:01.672399 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:01.672335    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v"
Sep 11 14:54:01.762712 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:01.762681    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24cf5ebe-57d5-41fc-a393-37d0a0bbfed5-plugins-config-volume\") pod \"24cf5ebe-57d5-41fc-a393-37d0a0bbfed5\" (UID: \"24cf5ebe-57d5-41fc-a393-37d0a0bbfed5\") "
Sep 11 14:54:01.762904 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:01.762861    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnfw4\" (UniqueName: \"kubernetes.io/projected/24cf5ebe-57d5-41fc-a393-37d0a0bbfed5-kube-api-access-qnfw4\") pod \"24cf5ebe-57d5-41fc-a393-37d0a0bbfed5\" (UID: \"24cf5ebe-57d5-41fc-a393-37d0a0bbfed5\") "
Sep 11 14:54:01.763180 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:01.763146    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24cf5ebe-57d5-41fc-a393-37d0a0bbfed5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24cf5ebe-57d5-41fc-a393-37d0a0bbfed5" (UID: "24cf5ebe-57d5-41fc-a393-37d0a0bbfed5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:54:01.766337 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:01.766306    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24cf5ebe-57d5-41fc-a393-37d0a0bbfed5-kube-api-access-qnfw4" (OuterVolumeSpecName: "kube-api-access-qnfw4") pod "24cf5ebe-57d5-41fc-a393-37d0a0bbfed5" (UID: "24cf5ebe-57d5-41fc-a393-37d0a0bbfed5"). InnerVolumeSpecName "kube-api-access-qnfw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:54:01.863575 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:01.863536    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnfw4\" (UniqueName: \"kubernetes.io/projected/24cf5ebe-57d5-41fc-a393-37d0a0bbfed5-kube-api-access-qnfw4\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:54:01.863575 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:01.863570    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24cf5ebe-57d5-41fc-a393-37d0a0bbfed5-plugins-config-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:54:02.581450 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:02.581415    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk"
Sep 11 14:54:02.581940 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:02.581874    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v"
Sep 11 14:54:02.639148 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:02.639114    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v"
Sep 11 14:54:02.639315 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:02.639118    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v" event={"ID":"24cf5ebe-57d5-41fc-a393-37d0a0bbfed5","Type":"ContainerDied","Data":"c579f368baeb77acaca9a2ce93cdc133a00760b18efa70c61795134ce9927ee4"}
Sep 11 14:54:02.639385 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:02.639313    2568 scope.go:117] "RemoveContainer" containerID="4b04bb24a5932bbe145c15aa017f7c6f343cfcd0722b3d9ac4c53de149cddff3"
Sep 11 14:54:02.664056 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:02.664023    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v"]
Sep 11 14:54:02.669981 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:02.669940    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f722bf59-7fcc65f7d6-6v88v"]
Sep 11 14:54:04.662393 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:04.662360    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24cf5ebe-57d5-41fc-a393-37d0a0bbfed5" path="/var/lib/kubelet/pods/24cf5ebe-57d5-41fc-a393-37d0a0bbfed5/volumes"
Sep 11 14:54:07.688465 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:07.688433    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8"]
Sep 11 14:54:07.688902 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:07.688888    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24cf5ebe-57d5-41fc-a393-37d0a0bbfed5" containerName="payload-pre-processing"
Sep 11 14:54:07.688902 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:07.688903    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="24cf5ebe-57d5-41fc-a393-37d0a0bbfed5" containerName="payload-pre-processing"
Sep 11 14:54:07.688992 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:07.688984    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="24cf5ebe-57d5-41fc-a393-37d0a0bbfed5" containerName="payload-pre-processing"
Sep 11 14:54:07.692263 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:07.692243    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8"
Sep 11 14:54:07.694903 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:07.694882    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4408ba1c-dockercfg-jp5jh\""
Sep 11 14:54:07.695025 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:07.694930    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4408ba1c\""
Sep 11 14:54:07.705004 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:07.704980    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8"]
Sep 11 14:54:07.832085 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:07.832041    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c96e61d-3440-447e-842b-a85cc6be23f3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8\" (UID: \"2c96e61d-3440-447e-842b-a85cc6be23f3\") " pod="openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8"
Sep 11 14:54:07.832252 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:07.832090    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbc8v\" (UniqueName: \"kubernetes.io/projected/2c96e61d-3440-447e-842b-a85cc6be23f3-kube-api-access-fbc8v\") pod \"payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8\" (UID: \"2c96e61d-3440-447e-842b-a85cc6be23f3\") " pod="openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8"
Sep 11 14:54:07.933569 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:07.933533    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c96e61d-3440-447e-842b-a85cc6be23f3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8\" (UID: \"2c96e61d-3440-447e-842b-a85cc6be23f3\") " pod="openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8"
Sep 11 14:54:07.933756 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:07.933576    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbc8v\" (UniqueName: \"kubernetes.io/projected/2c96e61d-3440-447e-842b-a85cc6be23f3-kube-api-access-fbc8v\") pod \"payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8\" (UID: \"2c96e61d-3440-447e-842b-a85cc6be23f3\") " pod="openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8"
Sep 11 14:54:07.934314 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:07.934288    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c96e61d-3440-447e-842b-a85cc6be23f3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8\" (UID: \"2c96e61d-3440-447e-842b-a85cc6be23f3\") " pod="openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8"
Sep 11 14:54:07.942393 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:07.942328    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbc8v\" (UniqueName: \"kubernetes.io/projected/2c96e61d-3440-447e-842b-a85cc6be23f3-kube-api-access-fbc8v\") pod \"payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8\" (UID: \"2c96e61d-3440-447e-842b-a85cc6be23f3\") " pod="openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8"
Sep 11 14:54:08.005309 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:08.005275    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8"
Sep 11 14:54:08.285491 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:08.285457    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8"]
Sep 11 14:54:08.287129 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:54:08.287103    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c96e61d_3440_447e_842b_a85cc6be23f3.slice/crio-fd2b9d6142f797ddc510405ca4cd685a9d41e6aee17ce056b556dd295ae8cde7 WatchSource:0}: Error finding container fd2b9d6142f797ddc510405ca4cd685a9d41e6aee17ce056b556dd295ae8cde7: Status 404 returned error can't find the container with id fd2b9d6142f797ddc510405ca4cd685a9d41e6aee17ce056b556dd295ae8cde7
Sep 11 14:54:08.673689 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:08.673598    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8" event={"ID":"2c96e61d-3440-447e-842b-a85cc6be23f3","Type":"ContainerStarted","Data":"04938841b1a37bc971db4b7b72206ec83baea99abf3e7313180b0b7e6e484db9"}
Sep 11 14:54:08.673689 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:08.673654    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8" event={"ID":"2c96e61d-3440-447e-842b-a85cc6be23f3","Type":"ContainerStarted","Data":"fd2b9d6142f797ddc510405ca4cd685a9d41e6aee17ce056b556dd295ae8cde7"}
Sep 11 14:54:08.673965 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:08.673793    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8"
Sep 11 14:54:08.692772 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:08.692726    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8" podStartSLOduration=1.6927110490000001 podStartE2EDuration="1.692711049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:54:08.691263921 +0000 UTC m=+828.479528223" watchObservedRunningTime="2026-09-11 14:54:08.692711049 +0000 UTC m=+828.480975347"
Sep 11 14:54:19.684726 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:19.684696    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8"
Sep 11 14:54:36.944324 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:36.944289    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8"]
Sep 11 14:54:36.944924 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:36.944493    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8" podUID="2c96e61d-3440-447e-842b-a85cc6be23f3" containerName="payload-processing" containerID="cri-o://04938841b1a37bc971db4b7b72206ec83baea99abf3e7313180b0b7e6e484db9" gracePeriod=30
Sep 11 14:54:37.201876 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:37.201800    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8"
Sep 11 14:54:37.327930 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:37.327895    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbc8v\" (UniqueName: \"kubernetes.io/projected/2c96e61d-3440-447e-842b-a85cc6be23f3-kube-api-access-fbc8v\") pod \"2c96e61d-3440-447e-842b-a85cc6be23f3\" (UID: \"2c96e61d-3440-447e-842b-a85cc6be23f3\") "
Sep 11 14:54:37.328103 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:37.327942    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c96e61d-3440-447e-842b-a85cc6be23f3-plugins-config-volume\") pod \"2c96e61d-3440-447e-842b-a85cc6be23f3\" (UID: \"2c96e61d-3440-447e-842b-a85cc6be23f3\") "
Sep 11 14:54:37.328394 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:37.328353    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c96e61d-3440-447e-842b-a85cc6be23f3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c96e61d-3440-447e-842b-a85cc6be23f3" (UID: "2c96e61d-3440-447e-842b-a85cc6be23f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:54:37.330025 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:37.330006    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c96e61d-3440-447e-842b-a85cc6be23f3-kube-api-access-fbc8v" (OuterVolumeSpecName: "kube-api-access-fbc8v") pod "2c96e61d-3440-447e-842b-a85cc6be23f3" (UID: "2c96e61d-3440-447e-842b-a85cc6be23f3"). InnerVolumeSpecName "kube-api-access-fbc8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:54:37.429260 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:37.429222    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbc8v\" (UniqueName: \"kubernetes.io/projected/2c96e61d-3440-447e-842b-a85cc6be23f3-kube-api-access-fbc8v\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:54:37.429260 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:37.429254    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c96e61d-3440-447e-842b-a85cc6be23f3-plugins-config-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:54:37.831730 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:37.831696    2568 generic.go:356] "Generic (PLEG): container finished" podID="2c96e61d-3440-447e-842b-a85cc6be23f3" containerID="04938841b1a37bc971db4b7b72206ec83baea99abf3e7313180b0b7e6e484db9" exitCode=0
Sep 11 14:54:37.831934 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:37.831742    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8" event={"ID":"2c96e61d-3440-447e-842b-a85cc6be23f3","Type":"ContainerDied","Data":"04938841b1a37bc971db4b7b72206ec83baea99abf3e7313180b0b7e6e484db9"}
Sep 11 14:54:37.831934 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:37.831749    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8"
Sep 11 14:54:37.831934 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:37.831781    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8" event={"ID":"2c96e61d-3440-447e-842b-a85cc6be23f3","Type":"ContainerDied","Data":"fd2b9d6142f797ddc510405ca4cd685a9d41e6aee17ce056b556dd295ae8cde7"}
Sep 11 14:54:37.831934 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:37.831802    2568 scope.go:117] "RemoveContainer" containerID="04938841b1a37bc971db4b7b72206ec83baea99abf3e7313180b0b7e6e484db9"
Sep 11 14:54:37.841104 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:37.841084    2568 scope.go:117] "RemoveContainer" containerID="04938841b1a37bc971db4b7b72206ec83baea99abf3e7313180b0b7e6e484db9"
Sep 11 14:54:37.841373 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:54:37.841356    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04938841b1a37bc971db4b7b72206ec83baea99abf3e7313180b0b7e6e484db9\": container with ID starting with 04938841b1a37bc971db4b7b72206ec83baea99abf3e7313180b0b7e6e484db9 not found: ID does not exist" containerID="04938841b1a37bc971db4b7b72206ec83baea99abf3e7313180b0b7e6e484db9"
Sep 11 14:54:37.841420 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:37.841383    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04938841b1a37bc971db4b7b72206ec83baea99abf3e7313180b0b7e6e484db9"} err="failed to get container status \"04938841b1a37bc971db4b7b72206ec83baea99abf3e7313180b0b7e6e484db9\": rpc error: code = NotFound desc = could not find container \"04938841b1a37bc971db4b7b72206ec83baea99abf3e7313180b0b7e6e484db9\": container with ID starting with 04938841b1a37bc971db4b7b72206ec83baea99abf3e7313180b0b7e6e484db9 not found: ID does not exist"
Sep 11 14:54:37.854598 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:37.854574    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8"]
Sep 11 14:54:37.860824 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:37.860795    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4408ba1c-b9665474b-54zz8"]
Sep 11 14:54:38.662018 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:38.661979    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c96e61d-3440-447e-842b-a85cc6be23f3" path="/var/lib/kubelet/pods/2c96e61d-3440-447e-842b-a85cc6be23f3/volumes"
Sep 11 14:54:43.465551 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:43.465516    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l"]
Sep 11 14:54:43.466298 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:43.466276    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c96e61d-3440-447e-842b-a85cc6be23f3" containerName="payload-processing"
Sep 11 14:54:43.466343 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:43.466303    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c96e61d-3440-447e-842b-a85cc6be23f3" containerName="payload-processing"
Sep 11 14:54:43.466423 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:43.466411    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c96e61d-3440-447e-842b-a85cc6be23f3" containerName="payload-processing"
Sep 11 14:54:43.471093 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:43.471059    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l"
Sep 11 14:54:43.474078 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:43.474056    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c0b7dbbd\""
Sep 11 14:54:43.474289 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:43.474270    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c0b7dbbd-dockercfg-dbpm8\""
Sep 11 14:54:43.479073 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:43.479014    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l"]
Sep 11 14:54:43.593431 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:43.593390    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4lpc\" (UniqueName: \"kubernetes.io/projected/3ce7e0b9-dfdc-43c0-85d2-1069c1910386-kube-api-access-n4lpc\") pod \"payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l\" (UID: \"3ce7e0b9-dfdc-43c0-85d2-1069c1910386\") " pod="openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l"
Sep 11 14:54:43.593431 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:43.593433    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce7e0b9-dfdc-43c0-85d2-1069c1910386-plugins-config-volume\") pod \"payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l\" (UID: \"3ce7e0b9-dfdc-43c0-85d2-1069c1910386\") " pod="openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l"
Sep 11 14:54:43.694926 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:43.694889    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4lpc\" (UniqueName: \"kubernetes.io/projected/3ce7e0b9-dfdc-43c0-85d2-1069c1910386-kube-api-access-n4lpc\") pod \"payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l\" (UID: \"3ce7e0b9-dfdc-43c0-85d2-1069c1910386\") " pod="openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l"
Sep 11 14:54:43.695114 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:43.694933    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce7e0b9-dfdc-43c0-85d2-1069c1910386-plugins-config-volume\") pod \"payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l\" (UID: \"3ce7e0b9-dfdc-43c0-85d2-1069c1910386\") " pod="openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l"
Sep 11 14:54:43.695639 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:43.695618    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce7e0b9-dfdc-43c0-85d2-1069c1910386-plugins-config-volume\") pod \"payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l\" (UID: \"3ce7e0b9-dfdc-43c0-85d2-1069c1910386\") " pod="openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l"
Sep 11 14:54:43.703245 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:43.703216    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4lpc\" (UniqueName: \"kubernetes.io/projected/3ce7e0b9-dfdc-43c0-85d2-1069c1910386-kube-api-access-n4lpc\") pod \"payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l\" (UID: \"3ce7e0b9-dfdc-43c0-85d2-1069c1910386\") " pod="openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l"
Sep 11 14:54:43.804990 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:43.804949    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l"
Sep 11 14:54:44.079781 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:44.079751    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l"]
Sep 11 14:54:44.081687 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:54:44.081659    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce7e0b9_dfdc_43c0_85d2_1069c1910386.slice/crio-9fe81c6cebe98bff248ca2a1109538bdfebbc018c816a8293ab4906901e440d7 WatchSource:0}: Error finding container 9fe81c6cebe98bff248ca2a1109538bdfebbc018c816a8293ab4906901e440d7: Status 404 returned error can't find the container with id 9fe81c6cebe98bff248ca2a1109538bdfebbc018c816a8293ab4906901e440d7
Sep 11 14:54:44.876178 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:44.876139    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l" event={"ID":"3ce7e0b9-dfdc-43c0-85d2-1069c1910386","Type":"ContainerStarted","Data":"d064fb907d15d0ea8abcb140a111557918b904bdb99c19f817c25bb02cbfbf73"}
Sep 11 14:54:44.876571 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:44.876252    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l"
Sep 11 14:54:44.876571 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:44.876272    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l" event={"ID":"3ce7e0b9-dfdc-43c0-85d2-1069c1910386","Type":"ContainerStarted","Data":"9fe81c6cebe98bff248ca2a1109538bdfebbc018c816a8293ab4906901e440d7"}
Sep 11 14:54:44.897589 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:44.897533    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l" podStartSLOduration=1.897518354 podStartE2EDuration="1.897518354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:54:44.895162363 +0000 UTC m=+864.683426673" watchObservedRunningTime="2026-09-11 14:54:44.897518354 +0000 UTC m=+864.685782652"
Sep 11 14:54:55.882913 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:54:55.882882    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l"
Sep 11 14:55:12.491812 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:12.491773    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l"]
Sep 11 14:55:12.492281 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:12.492083    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l" podUID="3ce7e0b9-dfdc-43c0-85d2-1069c1910386" containerName="payload-processing" containerID="cri-o://d064fb907d15d0ea8abcb140a111557918b904bdb99c19f817c25bb02cbfbf73" gracePeriod=30
Sep 11 14:55:12.747567 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:12.747507    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l"
Sep 11 14:55:12.879005 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:12.878971    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce7e0b9-dfdc-43c0-85d2-1069c1910386-plugins-config-volume\") pod \"3ce7e0b9-dfdc-43c0-85d2-1069c1910386\" (UID: \"3ce7e0b9-dfdc-43c0-85d2-1069c1910386\") "
Sep 11 14:55:12.879206 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:12.879021    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4lpc\" (UniqueName: \"kubernetes.io/projected/3ce7e0b9-dfdc-43c0-85d2-1069c1910386-kube-api-access-n4lpc\") pod \"3ce7e0b9-dfdc-43c0-85d2-1069c1910386\" (UID: \"3ce7e0b9-dfdc-43c0-85d2-1069c1910386\") "
Sep 11 14:55:12.879328 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:12.879303    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce7e0b9-dfdc-43c0-85d2-1069c1910386-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3ce7e0b9-dfdc-43c0-85d2-1069c1910386" (UID: "3ce7e0b9-dfdc-43c0-85d2-1069c1910386"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:55:12.881097 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:12.881074    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce7e0b9-dfdc-43c0-85d2-1069c1910386-kube-api-access-n4lpc" (OuterVolumeSpecName: "kube-api-access-n4lpc") pod "3ce7e0b9-dfdc-43c0-85d2-1069c1910386" (UID: "3ce7e0b9-dfdc-43c0-85d2-1069c1910386"). InnerVolumeSpecName "kube-api-access-n4lpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:55:12.980318 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:12.980238    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ce7e0b9-dfdc-43c0-85d2-1069c1910386-plugins-config-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:55:12.980318 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:12.980270    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4lpc\" (UniqueName: \"kubernetes.io/projected/3ce7e0b9-dfdc-43c0-85d2-1069c1910386-kube-api-access-n4lpc\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:55:13.028449 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:13.028410    2568 generic.go:356] "Generic (PLEG): container finished" podID="3ce7e0b9-dfdc-43c0-85d2-1069c1910386" containerID="d064fb907d15d0ea8abcb140a111557918b904bdb99c19f817c25bb02cbfbf73" exitCode=0
Sep 11 14:55:13.028645 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:13.028462    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l"
Sep 11 14:55:13.028645 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:13.028481    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l" event={"ID":"3ce7e0b9-dfdc-43c0-85d2-1069c1910386","Type":"ContainerDied","Data":"d064fb907d15d0ea8abcb140a111557918b904bdb99c19f817c25bb02cbfbf73"}
Sep 11 14:55:13.028645 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:13.028522    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l" event={"ID":"3ce7e0b9-dfdc-43c0-85d2-1069c1910386","Type":"ContainerDied","Data":"9fe81c6cebe98bff248ca2a1109538bdfebbc018c816a8293ab4906901e440d7"}
Sep 11 14:55:13.028645 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:13.028543    2568 scope.go:117] "RemoveContainer" containerID="d064fb907d15d0ea8abcb140a111557918b904bdb99c19f817c25bb02cbfbf73"
Sep 11 14:55:13.038038 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:13.038018    2568 scope.go:117] "RemoveContainer" containerID="d064fb907d15d0ea8abcb140a111557918b904bdb99c19f817c25bb02cbfbf73"
Sep 11 14:55:13.038350 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:55:13.038328    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d064fb907d15d0ea8abcb140a111557918b904bdb99c19f817c25bb02cbfbf73\": container with ID starting with d064fb907d15d0ea8abcb140a111557918b904bdb99c19f817c25bb02cbfbf73 not found: ID does not exist" containerID="d064fb907d15d0ea8abcb140a111557918b904bdb99c19f817c25bb02cbfbf73"
Sep 11 14:55:13.038394 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:13.038360    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d064fb907d15d0ea8abcb140a111557918b904bdb99c19f817c25bb02cbfbf73"} err="failed to get container status \"d064fb907d15d0ea8abcb140a111557918b904bdb99c19f817c25bb02cbfbf73\": rpc error: code = NotFound desc = could not find container \"d064fb907d15d0ea8abcb140a111557918b904bdb99c19f817c25bb02cbfbf73\": container with ID starting with d064fb907d15d0ea8abcb140a111557918b904bdb99c19f817c25bb02cbfbf73 not found: ID does not exist"
Sep 11 14:55:13.057875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:13.057844    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l"]
Sep 11 14:55:13.062554 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:13.062528    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c0b7dbbd-7dd99f844c-gc24l"]
Sep 11 14:55:14.662652 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:14.662621    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ce7e0b9-dfdc-43c0-85d2-1069c1910386" path="/var/lib/kubelet/pods/3ce7e0b9-dfdc-43c0-85d2-1069c1910386/volumes"
Sep 11 14:55:22.086362 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:22.086326    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"]
Sep 11 14:55:22.087139 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:22.087109    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ce7e0b9-dfdc-43c0-85d2-1069c1910386" containerName="payload-processing"
Sep 11 14:55:22.087139 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:22.087137    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ce7e0b9-dfdc-43c0-85d2-1069c1910386" containerName="payload-processing"
Sep 11 14:55:22.087302 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:22.087244    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ce7e0b9-dfdc-43c0-85d2-1069c1910386" containerName="payload-processing"
Sep 11 14:55:22.092682 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:22.092657    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"
Sep 11 14:55:22.095904 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:22.095874    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-58b751e1-serving-cert\""
Sep 11 14:55:22.099635 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:22.099608    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"]
Sep 11 14:55:22.173925 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:22.173883    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c367d1e2-9895-45ed-b58a-2c04e924ff41-metrics-certs\") pod \"maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6\" (UID: \"c367d1e2-9895-45ed-b58a-2c04e924ff41\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"
Sep 11 14:55:22.174128 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:22.173985    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c367d1e2-9895-45ed-b58a-2c04e924ff41-openshift-service-ca\") pod \"maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6\" (UID: \"c367d1e2-9895-45ed-b58a-2c04e924ff41\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"
Sep 11 14:55:22.174128 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:22.174009    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6xrb\" (UniqueName: \"kubernetes.io/projected/c367d1e2-9895-45ed-b58a-2c04e924ff41-kube-api-access-t6xrb\") pod \"maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6\" (UID: \"c367d1e2-9895-45ed-b58a-2c04e924ff41\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"
Sep 11 14:55:22.174128 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:22.174094    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c367d1e2-9895-45ed-b58a-2c04e924ff41-maas-api-tls\") pod \"maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6\" (UID: \"c367d1e2-9895-45ed-b58a-2c04e924ff41\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"
Sep 11 14:55:22.275426 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:22.275381    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c367d1e2-9895-45ed-b58a-2c04e924ff41-openshift-service-ca\") pod \"maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6\" (UID: \"c367d1e2-9895-45ed-b58a-2c04e924ff41\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"
Sep 11 14:55:22.275426 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:22.275429    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6xrb\" (UniqueName: \"kubernetes.io/projected/c367d1e2-9895-45ed-b58a-2c04e924ff41-kube-api-access-t6xrb\") pod \"maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6\" (UID: \"c367d1e2-9895-45ed-b58a-2c04e924ff41\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"
Sep 11 14:55:22.275685 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:22.275475    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c367d1e2-9895-45ed-b58a-2c04e924ff41-maas-api-tls\") pod \"maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6\" (UID: \"c367d1e2-9895-45ed-b58a-2c04e924ff41\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"
Sep 11 14:55:22.275685 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:22.275539    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c367d1e2-9895-45ed-b58a-2c04e924ff41-metrics-certs\") pod \"maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6\" (UID: \"c367d1e2-9895-45ed-b58a-2c04e924ff41\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"
Sep 11 14:55:22.275788 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:55:22.275684    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-serving-cert: secret "maas-api-e2e-derive-58b751e1-serving-cert" not found
Sep 11 14:55:22.275788 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:55:22.275772    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c367d1e2-9895-45ed-b58a-2c04e924ff41-maas-api-tls podName:c367d1e2-9895-45ed-b58a-2c04e924ff41 nodeName:}" failed. No retries permitted until 2026-09-11 14:55:22.775750064 +0000 UTC m=+902.564014351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c367d1e2-9895-45ed-b58a-2c04e924ff41-maas-api-tls") pod "maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6" (UID: "c367d1e2-9895-45ed-b58a-2c04e924ff41") : secret "maas-api-e2e-derive-58b751e1-serving-cert" not found
Sep 11 14:55:22.276162 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:22.276133    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c367d1e2-9895-45ed-b58a-2c04e924ff41-openshift-service-ca\") pod \"maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6\" (UID: \"c367d1e2-9895-45ed-b58a-2c04e924ff41\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"
Sep 11 14:55:22.277973 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:22.277954    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c367d1e2-9895-45ed-b58a-2c04e924ff41-metrics-certs\") pod \"maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6\" (UID: \"c367d1e2-9895-45ed-b58a-2c04e924ff41\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"
Sep 11 14:55:22.285556 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:22.285535    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6xrb\" (UniqueName: \"kubernetes.io/projected/c367d1e2-9895-45ed-b58a-2c04e924ff41-kube-api-access-t6xrb\") pod \"maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6\" (UID: \"c367d1e2-9895-45ed-b58a-2c04e924ff41\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"
Sep 11 14:55:22.780390 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:22.780358    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c367d1e2-9895-45ed-b58a-2c04e924ff41-maas-api-tls\") pod \"maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6\" (UID: \"c367d1e2-9895-45ed-b58a-2c04e924ff41\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"
Sep 11 14:55:22.782764 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:22.782736    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c367d1e2-9895-45ed-b58a-2c04e924ff41-maas-api-tls\") pod \"maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6\" (UID: \"c367d1e2-9895-45ed-b58a-2c04e924ff41\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"
Sep 11 14:55:23.007489 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:23.007455    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"
Sep 11 14:55:23.288486 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:23.288456    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"]
Sep 11 14:55:23.290224 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:55:23.290196    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc367d1e2_9895_45ed_b58a_2c04e924ff41.slice/crio-e8fbcc414e7807c0c82c9c878bb7705d8a9bcf5eb4a66099892bab72c9c78bb1 WatchSource:0}: Error finding container e8fbcc414e7807c0c82c9c878bb7705d8a9bcf5eb4a66099892bab72c9c78bb1: Status 404 returned error can't find the container with id e8fbcc414e7807c0c82c9c878bb7705d8a9bcf5eb4a66099892bab72c9c78bb1
Sep 11 14:55:24.092008 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:24.091977    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6" event={"ID":"c367d1e2-9895-45ed-b58a-2c04e924ff41","Type":"ContainerStarted","Data":"b277a3128336942e2722a769f4d11379d21d8414f3d4305c3f1cea0934f9816a"}
Sep 11 14:55:24.092194 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:24.092033    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6" event={"ID":"c367d1e2-9895-45ed-b58a-2c04e924ff41","Type":"ContainerStarted","Data":"e8fbcc414e7807c0c82c9c878bb7705d8a9bcf5eb4a66099892bab72c9c78bb1"}
Sep 11 14:55:24.092194 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:24.092062    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"
Sep 11 14:55:24.115413 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:24.115315    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6" podStartSLOduration=1.556060289 podStartE2EDuration="2.115298039s" totalImagesPullingTime="559.23775ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:55:24.113557137 +0000 UTC m=+903.901821437" watchObservedRunningTime="2026-09-11 14:55:24.115298039 +0000 UTC m=+903.903562337"
Sep 11 14:55:30.102641 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:55:30.102606    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"
Sep 11 14:56:00.203174 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:00.203141    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"]
Sep 11 14:56:00.203654 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:00.203387    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6" podUID="c367d1e2-9895-45ed-b58a-2c04e924ff41" containerName="maas-api" containerID="cri-o://b277a3128336942e2722a769f4d11379d21d8414f3d4305c3f1cea0934f9816a" gracePeriod=30
Sep 11 14:56:00.467372 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:00.467314    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"
Sep 11 14:56:00.543232 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:00.543197    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6xrb\" (UniqueName: \"kubernetes.io/projected/c367d1e2-9895-45ed-b58a-2c04e924ff41-kube-api-access-t6xrb\") pod \"c367d1e2-9895-45ed-b58a-2c04e924ff41\" (UID: \"c367d1e2-9895-45ed-b58a-2c04e924ff41\") "
Sep 11 14:56:00.543498 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:00.543467    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c367d1e2-9895-45ed-b58a-2c04e924ff41-maas-api-tls\") pod \"c367d1e2-9895-45ed-b58a-2c04e924ff41\" (UID: \"c367d1e2-9895-45ed-b58a-2c04e924ff41\") "
Sep 11 14:56:00.543630 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:00.543555    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c367d1e2-9895-45ed-b58a-2c04e924ff41-openshift-service-ca\") pod \"c367d1e2-9895-45ed-b58a-2c04e924ff41\" (UID: \"c367d1e2-9895-45ed-b58a-2c04e924ff41\") "
Sep 11 14:56:00.543630 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:00.543614    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c367d1e2-9895-45ed-b58a-2c04e924ff41-metrics-certs\") pod \"c367d1e2-9895-45ed-b58a-2c04e924ff41\" (UID: \"c367d1e2-9895-45ed-b58a-2c04e924ff41\") "
Sep 11 14:56:00.545410 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:00.545384    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c367d1e2-9895-45ed-b58a-2c04e924ff41-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c367d1e2-9895-45ed-b58a-2c04e924ff41" (UID: "c367d1e2-9895-45ed-b58a-2c04e924ff41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:56:00.559044 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:00.559001    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c367d1e2-9895-45ed-b58a-2c04e924ff41-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c367d1e2-9895-45ed-b58a-2c04e924ff41" (UID: "c367d1e2-9895-45ed-b58a-2c04e924ff41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:56:00.559264 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:00.559245    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c367d1e2-9895-45ed-b58a-2c04e924ff41-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c367d1e2-9895-45ed-b58a-2c04e924ff41" (UID: "c367d1e2-9895-45ed-b58a-2c04e924ff41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:56:00.561086 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:00.561018    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c367d1e2-9895-45ed-b58a-2c04e924ff41-kube-api-access-t6xrb" (OuterVolumeSpecName: "kube-api-access-t6xrb") pod "c367d1e2-9895-45ed-b58a-2c04e924ff41" (UID: "c367d1e2-9895-45ed-b58a-2c04e924ff41"). InnerVolumeSpecName "kube-api-access-t6xrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:56:00.645222 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:00.645187    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6xrb\" (UniqueName: \"kubernetes.io/projected/c367d1e2-9895-45ed-b58a-2c04e924ff41-kube-api-access-t6xrb\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:56:00.645222 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:00.645219    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c367d1e2-9895-45ed-b58a-2c04e924ff41-maas-api-tls\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:56:00.645222 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:00.645230    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c367d1e2-9895-45ed-b58a-2c04e924ff41-openshift-service-ca\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:56:00.645531 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:00.645238    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c367d1e2-9895-45ed-b58a-2c04e924ff41-metrics-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:56:01.289084 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:01.289046    2568 generic.go:356] "Generic (PLEG): container finished" podID="c367d1e2-9895-45ed-b58a-2c04e924ff41" containerID="b277a3128336942e2722a769f4d11379d21d8414f3d4305c3f1cea0934f9816a" exitCode=0
Sep 11 14:56:01.289506 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:01.289110    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"
Sep 11 14:56:01.289506 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:01.289132    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6" event={"ID":"c367d1e2-9895-45ed-b58a-2c04e924ff41","Type":"ContainerDied","Data":"b277a3128336942e2722a769f4d11379d21d8414f3d4305c3f1cea0934f9816a"}
Sep 11 14:56:01.289506 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:01.289194    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6" event={"ID":"c367d1e2-9895-45ed-b58a-2c04e924ff41","Type":"ContainerDied","Data":"e8fbcc414e7807c0c82c9c878bb7705d8a9bcf5eb4a66099892bab72c9c78bb1"}
Sep 11 14:56:01.289506 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:01.289223    2568 scope.go:117] "RemoveContainer" containerID="b277a3128336942e2722a769f4d11379d21d8414f3d4305c3f1cea0934f9816a"
Sep 11 14:56:01.298116 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:01.298098    2568 scope.go:117] "RemoveContainer" containerID="b277a3128336942e2722a769f4d11379d21d8414f3d4305c3f1cea0934f9816a"
Sep 11 14:56:01.298381 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:56:01.298361    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b277a3128336942e2722a769f4d11379d21d8414f3d4305c3f1cea0934f9816a\": container with ID starting with b277a3128336942e2722a769f4d11379d21d8414f3d4305c3f1cea0934f9816a not found: ID does not exist" containerID="b277a3128336942e2722a769f4d11379d21d8414f3d4305c3f1cea0934f9816a"
Sep 11 14:56:01.298450 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:01.298390    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b277a3128336942e2722a769f4d11379d21d8414f3d4305c3f1cea0934f9816a"} err="failed to get container status \"b277a3128336942e2722a769f4d11379d21d8414f3d4305c3f1cea0934f9816a\": rpc error: code = NotFound desc = could not find container \"b277a3128336942e2722a769f4d11379d21d8414f3d4305c3f1cea0934f9816a\": container with ID starting with b277a3128336942e2722a769f4d11379d21d8414f3d4305c3f1cea0934f9816a not found: ID does not exist"
Sep 11 14:56:01.313072 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:01.313048    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"]
Sep 11 14:56:01.322292 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:01.322252    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-58b751e1-7fb49f8b8b-r2tn6"]
Sep 11 14:56:02.663084 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:02.663052    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c367d1e2-9895-45ed-b58a-2c04e924ff41" path="/var/lib/kubelet/pods/c367d1e2-9895-45ed-b58a-2c04e924ff41/volumes"
Sep 11 14:56:12.549285 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:12.549241    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx"]
Sep 11 14:56:12.552956 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:12.549624    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx" podUID="22234bfd-e677-4897-b5a4-2194b9fe73dc" containerName="payload-processing" containerID="cri-o://6f752f7f978d6d1a66efc301aee3d455ea2a672496f994468dc08a6a3742e387" gracePeriod=30
Sep 11 14:56:12.818777 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:12.818753    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx"
Sep 11 14:56:12.857945 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:12.857917    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2k9h\" (UniqueName: \"kubernetes.io/projected/22234bfd-e677-4897-b5a4-2194b9fe73dc-kube-api-access-n2k9h\") pod \"22234bfd-e677-4897-b5a4-2194b9fe73dc\" (UID: \"22234bfd-e677-4897-b5a4-2194b9fe73dc\") "
Sep 11 14:56:12.858117 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:12.857997    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22234bfd-e677-4897-b5a4-2194b9fe73dc-plugins-config-volume\") pod \"22234bfd-e677-4897-b5a4-2194b9fe73dc\" (UID: \"22234bfd-e677-4897-b5a4-2194b9fe73dc\") "
Sep 11 14:56:12.859118 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:12.858418    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22234bfd-e677-4897-b5a4-2194b9fe73dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22234bfd-e677-4897-b5a4-2194b9fe73dc" (UID: "22234bfd-e677-4897-b5a4-2194b9fe73dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:56:12.864970 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:12.864926    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22234bfd-e677-4897-b5a4-2194b9fe73dc-kube-api-access-n2k9h" (OuterVolumeSpecName: "kube-api-access-n2k9h") pod "22234bfd-e677-4897-b5a4-2194b9fe73dc" (UID: "22234bfd-e677-4897-b5a4-2194b9fe73dc"). InnerVolumeSpecName "kube-api-access-n2k9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:56:12.959677 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:12.959642    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22234bfd-e677-4897-b5a4-2194b9fe73dc-plugins-config-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:56:12.959677 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:12.959673    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2k9h\" (UniqueName: \"kubernetes.io/projected/22234bfd-e677-4897-b5a4-2194b9fe73dc-kube-api-access-n2k9h\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:56:13.353925 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:13.353890    2568 generic.go:356] "Generic (PLEG): container finished" podID="22234bfd-e677-4897-b5a4-2194b9fe73dc" containerID="6f752f7f978d6d1a66efc301aee3d455ea2a672496f994468dc08a6a3742e387" exitCode=0
Sep 11 14:56:13.354094 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:13.353938    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx"
Sep 11 14:56:13.354094 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:13.353980    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx" event={"ID":"22234bfd-e677-4897-b5a4-2194b9fe73dc","Type":"ContainerDied","Data":"6f752f7f978d6d1a66efc301aee3d455ea2a672496f994468dc08a6a3742e387"}
Sep 11 14:56:13.354094 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:13.354040    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx" event={"ID":"22234bfd-e677-4897-b5a4-2194b9fe73dc","Type":"ContainerDied","Data":"59366daa914a0316eb1813fc8ceb551cb7a02074774bc49902ac39c001a4ec2e"}
Sep 11 14:56:13.354094 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:13.354067    2568 scope.go:117] "RemoveContainer" containerID="6f752f7f978d6d1a66efc301aee3d455ea2a672496f994468dc08a6a3742e387"
Sep 11 14:56:13.363569 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:13.363552    2568 scope.go:117] "RemoveContainer" containerID="6f752f7f978d6d1a66efc301aee3d455ea2a672496f994468dc08a6a3742e387"
Sep 11 14:56:13.363845 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:56:13.363810    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f752f7f978d6d1a66efc301aee3d455ea2a672496f994468dc08a6a3742e387\": container with ID starting with 6f752f7f978d6d1a66efc301aee3d455ea2a672496f994468dc08a6a3742e387 not found: ID does not exist" containerID="6f752f7f978d6d1a66efc301aee3d455ea2a672496f994468dc08a6a3742e387"
Sep 11 14:56:13.363925 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:13.363857    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f752f7f978d6d1a66efc301aee3d455ea2a672496f994468dc08a6a3742e387"} err="failed to get container status \"6f752f7f978d6d1a66efc301aee3d455ea2a672496f994468dc08a6a3742e387\": rpc error: code = NotFound desc = could not find container \"6f752f7f978d6d1a66efc301aee3d455ea2a672496f994468dc08a6a3742e387\": container with ID starting with 6f752f7f978d6d1a66efc301aee3d455ea2a672496f994468dc08a6a3742e387 not found: ID does not exist"
Sep 11 14:56:13.375623 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:13.375596    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx"]
Sep 11 14:56:13.382169 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:13.382148    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3219e3-7bd986c4ff-rkqqx"]
Sep 11 14:56:14.663100 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:56:14.663062    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22234bfd-e677-4897-b5a4-2194b9fe73dc" path="/var/lib/kubelet/pods/22234bfd-e677-4897-b5a4-2194b9fe73dc/volumes"
Sep 11 14:57:13.682135 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.682100    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"]
Sep 11 14:57:13.682721 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.682555    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c367d1e2-9895-45ed-b58a-2c04e924ff41" containerName="maas-api"
Sep 11 14:57:13.682721 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.682570    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c367d1e2-9895-45ed-b58a-2c04e924ff41" containerName="maas-api"
Sep 11 14:57:13.682721 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.682583    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22234bfd-e677-4897-b5a4-2194b9fe73dc" containerName="payload-processing"
Sep 11 14:57:13.682721 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.682591    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="22234bfd-e677-4897-b5a4-2194b9fe73dc" containerName="payload-processing"
Sep 11 14:57:13.682721 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.682675    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="22234bfd-e677-4897-b5a4-2194b9fe73dc" containerName="payload-processing"
Sep 11 14:57:13.682721 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.682684    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c367d1e2-9895-45ed-b58a-2c04e924ff41" containerName="maas-api"
Sep 11 14:57:13.686224 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.686196    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"
Sep 11 14:57:13.688763 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.688741    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-87d086-serving-cert\""
Sep 11 14:57:13.693272 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.693248    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"]
Sep 11 14:57:13.835437 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.835400    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-openshift-service-ca\") pod \"maas-api-e2e-explicit-87d086-566dddddb6-4tmzl\" (UID: \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"
Sep 11 14:57:13.835619 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.835597    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-metrics-certs\") pod \"maas-api-e2e-explicit-87d086-566dddddb6-4tmzl\" (UID: \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"
Sep 11 14:57:13.835688 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.835636    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv85v\" (UniqueName: \"kubernetes.io/projected/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-kube-api-access-pv85v\") pod \"maas-api-e2e-explicit-87d086-566dddddb6-4tmzl\" (UID: \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"
Sep 11 14:57:13.835798 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.835776    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-maas-api-tls\") pod \"maas-api-e2e-explicit-87d086-566dddddb6-4tmzl\" (UID: \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"
Sep 11 14:57:13.936977 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.936910    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-metrics-certs\") pod \"maas-api-e2e-explicit-87d086-566dddddb6-4tmzl\" (UID: \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"
Sep 11 14:57:13.936977 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.936942    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv85v\" (UniqueName: \"kubernetes.io/projected/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-kube-api-access-pv85v\") pod \"maas-api-e2e-explicit-87d086-566dddddb6-4tmzl\" (UID: \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"
Sep 11 14:57:13.936977 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.936963    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-maas-api-tls\") pod \"maas-api-e2e-explicit-87d086-566dddddb6-4tmzl\" (UID: \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"
Sep 11 14:57:13.937231 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.937008    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-openshift-service-ca\") pod \"maas-api-e2e-explicit-87d086-566dddddb6-4tmzl\" (UID: \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"
Sep 11 14:57:13.937231 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:57:13.937101    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-serving-cert: secret "maas-api-e2e-explicit-87d086-serving-cert" not found
Sep 11 14:57:13.937231 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:57:13.937191    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-maas-api-tls podName:4e4f2f95-ad12-4ede-b1c3-e17b6d6af489 nodeName:}" failed. No retries permitted until 2026-09-11 14:57:14.437168447 +0000 UTC m=+1014.225432740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-maas-api-tls") pod "maas-api-e2e-explicit-87d086-566dddddb6-4tmzl" (UID: "4e4f2f95-ad12-4ede-b1c3-e17b6d6af489") : secret "maas-api-e2e-explicit-87d086-serving-cert" not found
Sep 11 14:57:13.937693 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.937674    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-openshift-service-ca\") pod \"maas-api-e2e-explicit-87d086-566dddddb6-4tmzl\" (UID: \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"
Sep 11 14:57:13.939330 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.939313    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-metrics-certs\") pod \"maas-api-e2e-explicit-87d086-566dddddb6-4tmzl\" (UID: \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"
Sep 11 14:57:13.946583 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:13.946550    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv85v\" (UniqueName: \"kubernetes.io/projected/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-kube-api-access-pv85v\") pod \"maas-api-e2e-explicit-87d086-566dddddb6-4tmzl\" (UID: \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"
Sep 11 14:57:14.440216 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:14.440180    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-maas-api-tls\") pod \"maas-api-e2e-explicit-87d086-566dddddb6-4tmzl\" (UID: \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"
Sep 11 14:57:14.440394 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:57:14.440319    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-serving-cert: secret "maas-api-e2e-explicit-87d086-serving-cert" not found
Sep 11 14:57:14.440394 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:57:14.440389    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-maas-api-tls podName:4e4f2f95-ad12-4ede-b1c3-e17b6d6af489 nodeName:}" failed. No retries permitted until 2026-09-11 14:57:15.440373708 +0000 UTC m=+1015.228637985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-maas-api-tls") pod "maas-api-e2e-explicit-87d086-566dddddb6-4tmzl" (UID: "4e4f2f95-ad12-4ede-b1c3-e17b6d6af489") : secret "maas-api-e2e-explicit-87d086-serving-cert" not found
Sep 11 14:57:15.451004 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:15.450967    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-maas-api-tls\") pod \"maas-api-e2e-explicit-87d086-566dddddb6-4tmzl\" (UID: \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"
Sep 11 14:57:15.453426 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:15.453406    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-maas-api-tls\") pod \"maas-api-e2e-explicit-87d086-566dddddb6-4tmzl\" (UID: \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"
Sep 11 14:57:15.500330 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:15.500291    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"
Sep 11 14:57:15.779404 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:15.779378    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"]
Sep 11 14:57:15.781395 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:57:15.781364    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4f2f95_ad12_4ede_b1c3_e17b6d6af489.slice/crio-4e072c8a964f17122b771916c9bf3d177306f12f54f6f05431748efcfeecafe6 WatchSource:0}: Error finding container 4e072c8a964f17122b771916c9bf3d177306f12f54f6f05431748efcfeecafe6: Status 404 returned error can't find the container with id 4e072c8a964f17122b771916c9bf3d177306f12f54f6f05431748efcfeecafe6
Sep 11 14:57:15.782876 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:15.782860    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:57:16.690197 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:16.690122    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl" event={"ID":"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489","Type":"ContainerStarted","Data":"a7975cc7da9d0a4a774d73c1bb81de5f6e6739a773c85b0f7636efbd3758c8a2"}
Sep 11 14:57:16.690197 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:16.690171    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl" event={"ID":"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489","Type":"ContainerStarted","Data":"4e072c8a964f17122b771916c9bf3d177306f12f54f6f05431748efcfeecafe6"}
Sep 11 14:57:16.690565 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:16.690241    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"
Sep 11 14:57:16.712782 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:16.712737    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl" podStartSLOduration=3.058209005 podStartE2EDuration="3.712723942s" totalImagesPullingTime="654.514937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:57:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:57:16.710836652 +0000 UTC m=+1016.499100941" watchObservedRunningTime="2026-09-11 14:57:16.712723942 +0000 UTC m=+1016.500988296"
Sep 11 14:57:22.701303 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:22.701274    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"
Sep 11 14:57:42.286653 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.286617    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"]
Sep 11 14:57:42.290893 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.290869    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.294448 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.294420    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-577852b1-openshift-default-dockercfg-gw72j\""
Sep 11 14:57:42.308725 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.308693    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"]
Sep 11 14:57:42.402930 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.402898    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-credential-socket\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.402930 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.402934    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-workload-certs\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.403145 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.403062    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-podinfo\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.403145 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.403102    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-workload-socket\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.403226 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.403175    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-token\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.403226 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.403203    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cd73346-fc4e-4db9-8837-3442f9ac2605-istiod-ca-cert\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.403297 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.403239    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-envoy\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.403297 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.403282    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwrgz\" (UniqueName: \"kubernetes.io/projected/4cd73346-fc4e-4db9-8837-3442f9ac2605-kube-api-access-nwrgz\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.403371 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.403317    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-data\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.504835 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.504794    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-data\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.505017 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.504853    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-credential-socket\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.505017 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.504871    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-workload-certs\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.505017 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.504912    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-podinfo\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.505017 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.504930    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-workload-socket\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.505017 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.504962    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-token\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.505017 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.504989    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cd73346-fc4e-4db9-8837-3442f9ac2605-istiod-ca-cert\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.505340 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.505021    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-envoy\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.505340 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.505069    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwrgz\" (UniqueName: \"kubernetes.io/projected/4cd73346-fc4e-4db9-8837-3442f9ac2605-kube-api-access-nwrgz\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.505340 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.505276    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-data\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.505340 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.505329    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-workload-socket\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.505588 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.505567    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-workload-certs\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.505778 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.505745    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-credential-socket\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.506094 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.506070    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cd73346-fc4e-4db9-8837-3442f9ac2605-istiod-ca-cert\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.507380 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.507358    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-envoy\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.507515 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.507497    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-podinfo\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.513989 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.513966    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-token\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.514495 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.514469    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwrgz\" (UniqueName: \"kubernetes.io/projected/4cd73346-fc4e-4db9-8837-3442f9ac2605-kube-api-access-nwrgz\") pod \"e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") " pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.602055 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.601975    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:42.952494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.952280    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"]
Sep 11 14:57:42.955474 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:57:42.955447    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd73346_fc4e_4db9_8837_3442f9ac2605.slice/crio-449f873c66d7c3cbce53c4e836d4c6c7bf680630ccda70e410d2048831e40841 WatchSource:0}: Error finding container 449f873c66d7c3cbce53c4e836d4c6c7bf680630ccda70e410d2048831e40841: Status 404 returned error can't find the container with id 449f873c66d7c3cbce53c4e836d4c6c7bf680630ccda70e410d2048831e40841
Sep 11 14:57:42.958069 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.958030    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:57:42.958165 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.958102    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:57:42.958165 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:42.958133    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:57:43.832102 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:43.832070    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6" event={"ID":"4cd73346-fc4e-4db9-8837-3442f9ac2605","Type":"ContainerStarted","Data":"58901b450e0cab65b8b33e759bfffd44e7227e60a0f3662cc8f282991bf3a3fe"}
Sep 11 14:57:43.832461 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:43.832122    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6" event={"ID":"4cd73346-fc4e-4db9-8837-3442f9ac2605","Type":"ContainerStarted","Data":"449f873c66d7c3cbce53c4e836d4c6c7bf680630ccda70e410d2048831e40841"}
Sep 11 14:57:43.855065 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:43.855017    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6" podStartSLOduration=1.855000158 podStartE2EDuration="1.855000158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:57:43.853271012 +0000 UTC m=+1043.641535310" watchObservedRunningTime="2026-09-11 14:57:43.855000158 +0000 UTC m=+1043.643264465"
Sep 11 14:57:44.603157 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:44.603123    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:44.608447 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:44.608421    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:44.837550 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:44.837513    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:57:44.838470 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:57:44.838446    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:58:20.027528 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:20.027484    2568 generic.go:356] "Generic (PLEG): container finished" podID="4cd73346-fc4e-4db9-8837-3442f9ac2605" containerID="58901b450e0cab65b8b33e759bfffd44e7227e60a0f3662cc8f282991bf3a3fe" exitCode=0
Sep 11 14:58:20.028013 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:20.027536    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6" event={"ID":"4cd73346-fc4e-4db9-8837-3442f9ac2605","Type":"ContainerDied","Data":"58901b450e0cab65b8b33e759bfffd44e7227e60a0f3662cc8f282991bf3a3fe"}
Sep 11 14:58:20.028112 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:20.028066    2568 scope.go:117] "RemoveContainer" containerID="58901b450e0cab65b8b33e759bfffd44e7227e60a0f3662cc8f282991bf3a3fe"
Sep 11 14:58:20.029335 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:20.029301    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:58:20.029422 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:20.029370    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:58:20.029422 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:20.029403    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:58:21.039039 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:21.038992    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6" event={"ID":"4cd73346-fc4e-4db9-8837-3442f9ac2605","Type":"ContainerStarted","Data":"479d26163a372b36c549bc8a918df11dd0d687127a76c65ebca8a7421b7f6251"}
Sep 11 14:58:21.602266 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:21.602225    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:58:21.602463 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:21.602401    2568 patch_prober.go:28] interesting pod/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 11 14:58:21.602463 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:21.602453    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6" podUID="4cd73346-fc4e-4db9-8837-3442f9ac2605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 11 14:58:22.602845 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:22.602798    2568 patch_prober.go:28] interesting pod/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 11 14:58:22.603225 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:22.602873    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6" podUID="4cd73346-fc4e-4db9-8837-3442f9ac2605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 11 14:58:23.602312 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:23.602272    2568 patch_prober.go:28] interesting pod/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 11 14:58:23.602486 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:23.602346    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6" podUID="4cd73346-fc4e-4db9-8837-3442f9ac2605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 11 14:58:24.602417 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:24.602385    2568 patch_prober.go:28] interesting pod/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 11 14:58:24.602804 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:24.602448    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6" podUID="4cd73346-fc4e-4db9-8837-3442f9ac2605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 11 14:58:25.602662 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:25.602628    2568 patch_prober.go:28] interesting pod/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 11 14:58:25.603126 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:25.602690    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6" podUID="4cd73346-fc4e-4db9-8837-3442f9ac2605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 11 14:58:26.607859 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:26.607807    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:58:26.608229 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:26.608200    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:58:26.608901 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:26.608884    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:58:46.248136 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:46.248102    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"]
Sep 11 14:58:46.248560 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:46.248339    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6" podUID="4cd73346-fc4e-4db9-8837-3442f9ac2605" containerName="istio-proxy" containerID="cri-o://479d26163a372b36c549bc8a918df11dd0d687127a76c65ebca8a7421b7f6251" gracePeriod=30
Sep 11 14:58:51.514180 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.514125    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:58:51.581564 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.581533    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-envoy\") pod \"4cd73346-fc4e-4db9-8837-3442f9ac2605\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") "
Sep 11 14:58:51.581733 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.581574    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-credential-socket\") pod \"4cd73346-fc4e-4db9-8837-3442f9ac2605\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") "
Sep 11 14:58:51.581733 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.581611    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-data\") pod \"4cd73346-fc4e-4db9-8837-3442f9ac2605\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") "
Sep 11 14:58:51.581733 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.581642    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-workload-socket\") pod \"4cd73346-fc4e-4db9-8837-3442f9ac2605\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") "
Sep 11 14:58:51.581733 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.581661    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwrgz\" (UniqueName: \"kubernetes.io/projected/4cd73346-fc4e-4db9-8837-3442f9ac2605-kube-api-access-nwrgz\") pod \"4cd73346-fc4e-4db9-8837-3442f9ac2605\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") "
Sep 11 14:58:51.581733 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.581686    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-workload-certs\") pod \"4cd73346-fc4e-4db9-8837-3442f9ac2605\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") "
Sep 11 14:58:51.581733 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.581713    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-podinfo\") pod \"4cd73346-fc4e-4db9-8837-3442f9ac2605\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") "
Sep 11 14:58:51.582118 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.581782    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-token\") pod \"4cd73346-fc4e-4db9-8837-3442f9ac2605\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") "
Sep 11 14:58:51.582118 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.581826    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cd73346-fc4e-4db9-8837-3442f9ac2605-istiod-ca-cert\") pod \"4cd73346-fc4e-4db9-8837-3442f9ac2605\" (UID: \"4cd73346-fc4e-4db9-8837-3442f9ac2605\") "
Sep 11 14:58:51.582118 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.582015    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4cd73346-fc4e-4db9-8837-3442f9ac2605" (UID: "4cd73346-fc4e-4db9-8837-3442f9ac2605"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:58:51.582118 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.582089    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-data" (OuterVolumeSpecName: "istio-data") pod "4cd73346-fc4e-4db9-8837-3442f9ac2605" (UID: "4cd73346-fc4e-4db9-8837-3442f9ac2605"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:58:51.582383 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.582247    2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-credential-socket\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:58:51.582383 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.582265    2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-data\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:58:51.582383 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.582290    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4cd73346-fc4e-4db9-8837-3442f9ac2605" (UID: "4cd73346-fc4e-4db9-8837-3442f9ac2605"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:58:51.582383 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.582342    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd73346-fc4e-4db9-8837-3442f9ac2605-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4cd73346-fc4e-4db9-8837-3442f9ac2605" (UID: "4cd73346-fc4e-4db9-8837-3442f9ac2605"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:58:51.582604 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.582392    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4cd73346-fc4e-4db9-8837-3442f9ac2605" (UID: "4cd73346-fc4e-4db9-8837-3442f9ac2605"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:58:51.583847 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.583806    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4cd73346-fc4e-4db9-8837-3442f9ac2605" (UID: "4cd73346-fc4e-4db9-8837-3442f9ac2605"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:58:51.584047 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.584030    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4cd73346-fc4e-4db9-8837-3442f9ac2605" (UID: "4cd73346-fc4e-4db9-8837-3442f9ac2605"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:58:51.584107 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.584050    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-token" (OuterVolumeSpecName: "istio-token") pod "4cd73346-fc4e-4db9-8837-3442f9ac2605" (UID: "4cd73346-fc4e-4db9-8837-3442f9ac2605"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:58:51.584150 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.584114    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd73346-fc4e-4db9-8837-3442f9ac2605-kube-api-access-nwrgz" (OuterVolumeSpecName: "kube-api-access-nwrgz") pod "4cd73346-fc4e-4db9-8837-3442f9ac2605" (UID: "4cd73346-fc4e-4db9-8837-3442f9ac2605"). InnerVolumeSpecName "kube-api-access-nwrgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:58:51.682956 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.682928    2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-envoy\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:58:51.682956 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.682959    2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-workload-socket\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:58:51.683102 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.682969    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwrgz\" (UniqueName: \"kubernetes.io/projected/4cd73346-fc4e-4db9-8837-3442f9ac2605-kube-api-access-nwrgz\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:58:51.683102 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.682979    2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cd73346-fc4e-4db9-8837-3442f9ac2605-workload-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:58:51.683102 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.682987    2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-podinfo\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:58:51.683102 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.682995    2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cd73346-fc4e-4db9-8837-3442f9ac2605-istio-token\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:58:51.683102 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:51.683003    2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cd73346-fc4e-4db9-8837-3442f9ac2605-istiod-ca-cert\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:58:52.213346 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:52.213311    2568 generic.go:356] "Generic (PLEG): container finished" podID="4cd73346-fc4e-4db9-8837-3442f9ac2605" containerID="479d26163a372b36c549bc8a918df11dd0d687127a76c65ebca8a7421b7f6251" exitCode=0
Sep 11 14:58:52.213528 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:52.213391    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6" event={"ID":"4cd73346-fc4e-4db9-8837-3442f9ac2605","Type":"ContainerDied","Data":"479d26163a372b36c549bc8a918df11dd0d687127a76c65ebca8a7421b7f6251"}
Sep 11 14:58:52.213528 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:52.213394    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"
Sep 11 14:58:52.213528 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:52.213443    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6" event={"ID":"4cd73346-fc4e-4db9-8837-3442f9ac2605","Type":"ContainerDied","Data":"449f873c66d7c3cbce53c4e836d4c6c7bf680630ccda70e410d2048831e40841"}
Sep 11 14:58:52.213528 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:52.213471    2568 scope.go:117] "RemoveContainer" containerID="479d26163a372b36c549bc8a918df11dd0d687127a76c65ebca8a7421b7f6251"
Sep 11 14:58:52.223619 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:52.223601    2568 scope.go:117] "RemoveContainer" containerID="58901b450e0cab65b8b33e759bfffd44e7227e60a0f3662cc8f282991bf3a3fe"
Sep 11 14:58:52.232506 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:52.232487    2568 scope.go:117] "RemoveContainer" containerID="479d26163a372b36c549bc8a918df11dd0d687127a76c65ebca8a7421b7f6251"
Sep 11 14:58:52.232755 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:58:52.232735    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"479d26163a372b36c549bc8a918df11dd0d687127a76c65ebca8a7421b7f6251\": container with ID starting with 479d26163a372b36c549bc8a918df11dd0d687127a76c65ebca8a7421b7f6251 not found: ID does not exist" containerID="479d26163a372b36c549bc8a918df11dd0d687127a76c65ebca8a7421b7f6251"
Sep 11 14:58:52.232836 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:52.232763    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"479d26163a372b36c549bc8a918df11dd0d687127a76c65ebca8a7421b7f6251"} err="failed to get container status \"479d26163a372b36c549bc8a918df11dd0d687127a76c65ebca8a7421b7f6251\": rpc error: code = NotFound desc = could not find container \"479d26163a372b36c549bc8a918df11dd0d687127a76c65ebca8a7421b7f6251\": container with ID starting with 479d26163a372b36c549bc8a918df11dd0d687127a76c65ebca8a7421b7f6251 not found: ID does not exist"
Sep 11 14:58:52.232836 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:52.232782    2568 scope.go:117] "RemoveContainer" containerID="58901b450e0cab65b8b33e759bfffd44e7227e60a0f3662cc8f282991bf3a3fe"
Sep 11 14:58:52.233047 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:58:52.233030    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58901b450e0cab65b8b33e759bfffd44e7227e60a0f3662cc8f282991bf3a3fe\": container with ID starting with 58901b450e0cab65b8b33e759bfffd44e7227e60a0f3662cc8f282991bf3a3fe not found: ID does not exist" containerID="58901b450e0cab65b8b33e759bfffd44e7227e60a0f3662cc8f282991bf3a3fe"
Sep 11 14:58:52.233096 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:52.233051    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58901b450e0cab65b8b33e759bfffd44e7227e60a0f3662cc8f282991bf3a3fe"} err="failed to get container status \"58901b450e0cab65b8b33e759bfffd44e7227e60a0f3662cc8f282991bf3a3fe\": rpc error: code = NotFound desc = could not find container \"58901b450e0cab65b8b33e759bfffd44e7227e60a0f3662cc8f282991bf3a3fe\": container with ID starting with 58901b450e0cab65b8b33e759bfffd44e7227e60a0f3662cc8f282991bf3a3fe not found: ID does not exist"
Sep 11 14:58:52.239211 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:52.239190    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"]
Sep 11 14:58:52.243553 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:52.243531    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-577852b1-openshift-default-6c6fd5f5b4-zb6j6"]
Sep 11 14:58:52.661789 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:58:52.661743    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cd73346-fc4e-4db9-8837-3442f9ac2605" path="/var/lib/kubelet/pods/4cd73346-fc4e-4db9-8837-3442f9ac2605/volumes"
Sep 11 14:59:00.102923 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:00.102887    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"]
Sep 11 14:59:00.103392 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:00.103276    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl" podUID="4e4f2f95-ad12-4ede-b1c3-e17b6d6af489" containerName="maas-api" containerID="cri-o://a7975cc7da9d0a4a774d73c1bb81de5f6e6739a773c85b0f7636efbd3758c8a2" gracePeriod=30
Sep 11 14:59:00.256466 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:00.256434    2568 generic.go:356] "Generic (PLEG): container finished" podID="4e4f2f95-ad12-4ede-b1c3-e17b6d6af489" containerID="a7975cc7da9d0a4a774d73c1bb81de5f6e6739a773c85b0f7636efbd3758c8a2" exitCode=0
Sep 11 14:59:00.256608 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:00.256481    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl" event={"ID":"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489","Type":"ContainerDied","Data":"a7975cc7da9d0a4a774d73c1bb81de5f6e6739a773c85b0f7636efbd3758c8a2"}
Sep 11 14:59:00.365181 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:00.365123    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"
Sep 11 14:59:00.468088 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:00.468057    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv85v\" (UniqueName: \"kubernetes.io/projected/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-kube-api-access-pv85v\") pod \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\" (UID: \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\") "
Sep 11 14:59:00.468266 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:00.468104    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-metrics-certs\") pod \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\" (UID: \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\") "
Sep 11 14:59:00.468266 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:00.468159    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-openshift-service-ca\") pod \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\" (UID: \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\") "
Sep 11 14:59:00.468266 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:00.468179    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-maas-api-tls\") pod \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\" (UID: \"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489\") "
Sep 11 14:59:00.468573 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:00.468544    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4e4f2f95-ad12-4ede-b1c3-e17b6d6af489" (UID: "4e4f2f95-ad12-4ede-b1c3-e17b6d6af489"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:59:00.470414 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:00.470391    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4e4f2f95-ad12-4ede-b1c3-e17b6d6af489" (UID: "4e4f2f95-ad12-4ede-b1c3-e17b6d6af489"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:59:00.470516 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:00.470467    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4e4f2f95-ad12-4ede-b1c3-e17b6d6af489" (UID: "4e4f2f95-ad12-4ede-b1c3-e17b6d6af489"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:59:00.470516 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:00.470488    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-kube-api-access-pv85v" (OuterVolumeSpecName: "kube-api-access-pv85v") pod "4e4f2f95-ad12-4ede-b1c3-e17b6d6af489" (UID: "4e4f2f95-ad12-4ede-b1c3-e17b6d6af489"). InnerVolumeSpecName "kube-api-access-pv85v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:59:00.569488 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:00.569455    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv85v\" (UniqueName: \"kubernetes.io/projected/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-kube-api-access-pv85v\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:00.569488 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:00.569483    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-metrics-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:00.569488 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:00.569492    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-openshift-service-ca\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:00.569802 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:00.569500    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489-maas-api-tls\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:01.263432 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:01.263400    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"
Sep 11 14:59:01.263904 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:01.263437    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl" event={"ID":"4e4f2f95-ad12-4ede-b1c3-e17b6d6af489","Type":"ContainerDied","Data":"4e072c8a964f17122b771916c9bf3d177306f12f54f6f05431748efcfeecafe6"}
Sep 11 14:59:01.263904 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:01.263499    2568 scope.go:117] "RemoveContainer" containerID="a7975cc7da9d0a4a774d73c1bb81de5f6e6739a773c85b0f7636efbd3758c8a2"
Sep 11 14:59:01.286120 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:01.286092    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"]
Sep 11 14:59:01.290792 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:01.290771    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-87d086-566dddddb6-4tmzl"]
Sep 11 14:59:02.662565 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.662519    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e4f2f95-ad12-4ede-b1c3-e17b6d6af489" path="/var/lib/kubelet/pods/4e4f2f95-ad12-4ede-b1c3-e17b6d6af489/volumes"
Sep 11 14:59:02.891225 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.891188    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"]
Sep 11 14:59:02.891654 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.891640    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cd73346-fc4e-4db9-8837-3442f9ac2605" containerName="istio-proxy"
Sep 11 14:59:02.891697 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.891655    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cd73346-fc4e-4db9-8837-3442f9ac2605" containerName="istio-proxy"
Sep 11 14:59:02.891697 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.891665    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cd73346-fc4e-4db9-8837-3442f9ac2605" containerName="istio-proxy"
Sep 11 14:59:02.891697 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.891671    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cd73346-fc4e-4db9-8837-3442f9ac2605" containerName="istio-proxy"
Sep 11 14:59:02.891697 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.891687    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e4f2f95-ad12-4ede-b1c3-e17b6d6af489" containerName="maas-api"
Sep 11 14:59:02.891697 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.891694    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e4f2f95-ad12-4ede-b1c3-e17b6d6af489" containerName="maas-api"
Sep 11 14:59:02.891871 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.891766    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cd73346-fc4e-4db9-8837-3442f9ac2605" containerName="istio-proxy"
Sep 11 14:59:02.891871 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.891780    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e4f2f95-ad12-4ede-b1c3-e17b6d6af489" containerName="maas-api"
Sep 11 14:59:02.896255 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.896237    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:02.898933 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.898909    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-c4e39b-openshift-default-dockercfg-hd5kh\""
Sep 11 14:59:02.916738 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.916667    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"]
Sep 11 14:59:02.991255 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.991214    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-envoy\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:02.991433 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.991261    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e90696-abd3-4ccd-b528-df0eea3b2023-istiod-ca-cert\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:02.991433 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.991359    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-workload-socket\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:02.991433 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.991394    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsq57\" (UniqueName: \"kubernetes.io/projected/51e90696-abd3-4ccd-b528-df0eea3b2023-kube-api-access-tsq57\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:02.991433 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.991415    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-credential-socket\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:02.991578 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.991483    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-token\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:02.991578 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.991530    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-podinfo\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:02.991578 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.991569    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-data\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:02.991693 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:02.991627    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-workload-certs\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:03.092607 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.092570    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-envoy\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:03.092866 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.092659    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e90696-abd3-4ccd-b528-df0eea3b2023-istiod-ca-cert\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:03.093066 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.093047    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-workload-socket\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:03.093224 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.093187    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsq57\" (UniqueName: \"kubernetes.io/projected/51e90696-abd3-4ccd-b528-df0eea3b2023-kube-api-access-tsq57\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:03.093333 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.093228    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-credential-socket\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:03.093333 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.093260    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-token\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:03.093333 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.093287    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-podinfo\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:03.093494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.093333    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-data\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:03.093494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.093397    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-workload-certs\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:03.093494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.093391    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-workload-socket\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:03.093494 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.093438    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e90696-abd3-4ccd-b528-df0eea3b2023-istiod-ca-cert\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:03.093694 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.093583    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-credential-socket\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:03.093694 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.093665    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-workload-certs\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:03.093922 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.093876    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-data\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:03.095732 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.095701    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-envoy\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:03.096408 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.096390    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-podinfo\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:03.107913 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.107884    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsq57\" (UniqueName: \"kubernetes.io/projected/51e90696-abd3-4ccd-b528-df0eea3b2023-kube-api-access-tsq57\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:03.108001 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.107952    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-token\") pod \"e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") " pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:03.206676 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.206595    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:03.491891 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.491385    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"]
Sep 11 14:59:03.498499 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:59:03.498469    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e90696_abd3_4ccd_b528_df0eea3b2023.slice/crio-a2080fc07de3ec06d186aab28d3ce77e830c85899964411242567d40699fefd8 WatchSource:0}: Error finding container a2080fc07de3ec06d186aab28d3ce77e830c85899964411242567d40699fefd8: Status 404 returned error can't find the container with id a2080fc07de3ec06d186aab28d3ce77e830c85899964411242567d40699fefd8
Sep 11 14:59:03.500660 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.500626    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:59:03.500717 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.500692    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:59:03.500753 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:03.500720    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:59:04.283648 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:04.283612    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph" event={"ID":"51e90696-abd3-4ccd-b528-df0eea3b2023","Type":"ContainerStarted","Data":"bdc9afc25096dcdb57f73995eea5a007211ddac2d837e53c477bb4671bc40491"}
Sep 11 14:59:04.284066 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:04.283663    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph" event={"ID":"51e90696-abd3-4ccd-b528-df0eea3b2023","Type":"ContainerStarted","Data":"a2080fc07de3ec06d186aab28d3ce77e830c85899964411242567d40699fefd8"}
Sep 11 14:59:04.311104 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:04.311058    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph" podStartSLOduration=2.311043556 podStartE2EDuration="2.311043556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:59:04.307429749 +0000 UTC m=+1124.095694058" watchObservedRunningTime="2026-09-11 14:59:04.311043556 +0000 UTC m=+1124.099307867"
Sep 11 14:59:05.207035 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:05.206999    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:05.212160 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:05.212130    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:05.289241 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:05.289206    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:05.290087 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:05.290068    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:10.776418 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:10.776386    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr"]
Sep 11 14:59:10.776912 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:10.776898    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cd73346-fc4e-4db9-8837-3442f9ac2605" containerName="istio-proxy"
Sep 11 14:59:10.780326 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:10.780310    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr"
Sep 11 14:59:10.783025 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:10.783005    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-c4e39b-dockercfg-rm65r\""
Sep 11 14:59:10.783122 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:10.783089    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c4e39b\""
Sep 11 14:59:10.790992 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:10.790966    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr"]
Sep 11 14:59:10.870320 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:10.870285    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbcdde6-9675-486d-b368-2aef2b6bed05-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr\" (UID: \"2dbcdde6-9675-486d-b368-2aef2b6bed05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr"
Sep 11 14:59:10.870467 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:10.870346    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7md7\" (UniqueName: \"kubernetes.io/projected/2dbcdde6-9675-486d-b368-2aef2b6bed05-kube-api-access-v7md7\") pod \"payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr\" (UID: \"2dbcdde6-9675-486d-b368-2aef2b6bed05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr"
Sep 11 14:59:10.971837 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:10.971682    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbcdde6-9675-486d-b368-2aef2b6bed05-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr\" (UID: \"2dbcdde6-9675-486d-b368-2aef2b6bed05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr"
Sep 11 14:59:10.971837 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:10.971791    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7md7\" (UniqueName: \"kubernetes.io/projected/2dbcdde6-9675-486d-b368-2aef2b6bed05-kube-api-access-v7md7\") pod \"payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr\" (UID: \"2dbcdde6-9675-486d-b368-2aef2b6bed05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr"
Sep 11 14:59:10.973201 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:10.973164    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbcdde6-9675-486d-b368-2aef2b6bed05-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr\" (UID: \"2dbcdde6-9675-486d-b368-2aef2b6bed05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr"
Sep 11 14:59:10.982667 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:10.982635    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7md7\" (UniqueName: \"kubernetes.io/projected/2dbcdde6-9675-486d-b368-2aef2b6bed05-kube-api-access-v7md7\") pod \"payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr\" (UID: \"2dbcdde6-9675-486d-b368-2aef2b6bed05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr"
Sep 11 14:59:11.092337 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:11.092257    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr"
Sep 11 14:59:11.371665 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:11.371636    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr"]
Sep 11 14:59:11.373523 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:59:11.373490    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbcdde6_9675_486d_b368_2aef2b6bed05.slice/crio-39141a37ceb35de82a2b7feb25d5c81aa67337fa8c655f536c30584a2d02a52d WatchSource:0}: Error finding container 39141a37ceb35de82a2b7feb25d5c81aa67337fa8c655f536c30584a2d02a52d: Status 404 returned error can't find the container with id 39141a37ceb35de82a2b7feb25d5c81aa67337fa8c655f536c30584a2d02a52d
Sep 11 14:59:12.329809 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:12.329766    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr" event={"ID":"2dbcdde6-9675-486d-b368-2aef2b6bed05","Type":"ContainerStarted","Data":"494506fba4ebbf30aa46c83015613ca2bdaa7002616edf16859a90384de5eab5"}
Sep 11 14:59:12.329809 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:12.329840    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr" event={"ID":"2dbcdde6-9675-486d-b368-2aef2b6bed05","Type":"ContainerStarted","Data":"39141a37ceb35de82a2b7feb25d5c81aa67337fa8c655f536c30584a2d02a52d"}
Sep 11 14:59:12.330277 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:12.329863    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr"
Sep 11 14:59:12.348451 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:12.348398    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr" podStartSLOduration=2.348384572 podStartE2EDuration="2.348384572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:59:12.346643317 +0000 UTC m=+1132.134907616" watchObservedRunningTime="2026-09-11 14:59:12.348384572 +0000 UTC m=+1132.136648870"
Sep 11 14:59:23.338168 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:23.338138    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr"
Sep 11 14:59:31.486210 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:31.486176    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"]
Sep 11 14:59:31.491645 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:31.491617    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"
Sep 11 14:59:31.494621 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:31.494596    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-c0e28e-serving-cert\""
Sep 11 14:59:31.505134 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:31.505107    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"]
Sep 11 14:59:31.558035 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:31.557995    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0abfb50a-d22e-4dc4-b8af-814179a37f9c-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv\" (UID: \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"
Sep 11 14:59:31.558225 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:31.558043    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0abfb50a-d22e-4dc4-b8af-814179a37f9c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv\" (UID: \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"
Sep 11 14:59:31.558225 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:31.558147    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0abfb50a-d22e-4dc4-b8af-814179a37f9c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv\" (UID: \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"
Sep 11 14:59:31.558327 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:31.558239    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlbss\" (UniqueName: \"kubernetes.io/projected/0abfb50a-d22e-4dc4-b8af-814179a37f9c-kube-api-access-mlbss\") pod \"maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv\" (UID: \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"
Sep 11 14:59:31.659388 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:31.659355    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlbss\" (UniqueName: \"kubernetes.io/projected/0abfb50a-d22e-4dc4-b8af-814179a37f9c-kube-api-access-mlbss\") pod \"maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv\" (UID: \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"
Sep 11 14:59:31.659549 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:31.659423    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0abfb50a-d22e-4dc4-b8af-814179a37f9c-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv\" (UID: \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"
Sep 11 14:59:31.659549 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:31.659458    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0abfb50a-d22e-4dc4-b8af-814179a37f9c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv\" (UID: \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"
Sep 11 14:59:31.659549 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:31.659538    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0abfb50a-d22e-4dc4-b8af-814179a37f9c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv\" (UID: \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"
Sep 11 14:59:31.659678 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:59:31.659657    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-serving-cert: secret "maas-api-e2e-shared-b-w2-c0e28e-serving-cert" not found
Sep 11 14:59:31.659732 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:59:31.659721    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0abfb50a-d22e-4dc4-b8af-814179a37f9c-maas-api-tls podName:0abfb50a-d22e-4dc4-b8af-814179a37f9c nodeName:}" failed. No retries permitted until 2026-09-11 14:59:32.159700582 +0000 UTC m=+1151.947964868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0abfb50a-d22e-4dc4-b8af-814179a37f9c-maas-api-tls") pod "maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv" (UID: "0abfb50a-d22e-4dc4-b8af-814179a37f9c") : secret "maas-api-e2e-shared-b-w2-c0e28e-serving-cert" not found
Sep 11 14:59:31.660243 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:31.660219    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0abfb50a-d22e-4dc4-b8af-814179a37f9c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv\" (UID: \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"
Sep 11 14:59:31.662072 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:31.662046    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0abfb50a-d22e-4dc4-b8af-814179a37f9c-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv\" (UID: \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"
Sep 11 14:59:31.670685 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:31.670662    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlbss\" (UniqueName: \"kubernetes.io/projected/0abfb50a-d22e-4dc4-b8af-814179a37f9c-kube-api-access-mlbss\") pod \"maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv\" (UID: \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"
Sep 11 14:59:32.163602 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:32.163571    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0abfb50a-d22e-4dc4-b8af-814179a37f9c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv\" (UID: \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"
Sep 11 14:59:32.166056 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:32.166032    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0abfb50a-d22e-4dc4-b8af-814179a37f9c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv\" (UID: \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"
Sep 11 14:59:32.405317 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:32.405277    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"
Sep 11 14:59:32.705806 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:32.705710    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"]
Sep 11 14:59:32.708831 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:59:32.708724    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0abfb50a_d22e_4dc4_b8af_814179a37f9c.slice/crio-77ed6ad4c7384b63035e67e42a579efee04b7518673265a77a5a39c4ece77df2 WatchSource:0}: Error finding container 77ed6ad4c7384b63035e67e42a579efee04b7518673265a77a5a39c4ece77df2: Status 404 returned error can't find the container with id 77ed6ad4c7384b63035e67e42a579efee04b7518673265a77a5a39c4ece77df2
Sep 11 14:59:33.456891 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:33.456780    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv" event={"ID":"0abfb50a-d22e-4dc4-b8af-814179a37f9c","Type":"ContainerStarted","Data":"f6e3b24c1a8f30d0b5ce1c40f2fbef01e39ca4312e8eab2b2601c54380148a1c"}
Sep 11 14:59:33.456891 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:33.456872    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv" event={"ID":"0abfb50a-d22e-4dc4-b8af-814179a37f9c","Type":"ContainerStarted","Data":"77ed6ad4c7384b63035e67e42a579efee04b7518673265a77a5a39c4ece77df2"}
Sep 11 14:59:33.457111 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:33.456922    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"
Sep 11 14:59:33.475654 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:33.475600    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv" podStartSLOduration=2.127943076 podStartE2EDuration="2.475582407s" totalImagesPullingTime="347.639331ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:59:33.473957584 +0000 UTC m=+1153.262221879" watchObservedRunningTime="2026-09-11 14:59:33.475582407 +0000 UTC m=+1153.263846706"
Sep 11 14:59:34.147875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:34.145611    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"]
Sep 11 14:59:34.147875 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:34.145913    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56" podUID="2daa6ec8-8f7b-4532-a7a5-192cd9df0793" containerName="maas-api" containerID="cri-o://bb5ff971effee8ad3a3db06956bf92e50299e7fedb408f624d5e7c4156fc72a0" gracePeriod=30
Sep 11 14:59:34.465408 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:34.465374    2568 generic.go:356] "Generic (PLEG): container finished" podID="2daa6ec8-8f7b-4532-a7a5-192cd9df0793" containerID="bb5ff971effee8ad3a3db06956bf92e50299e7fedb408f624d5e7c4156fc72a0" exitCode=0
Sep 11 14:59:34.465561 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:34.465456    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56" event={"ID":"2daa6ec8-8f7b-4532-a7a5-192cd9df0793","Type":"ContainerDied","Data":"bb5ff971effee8ad3a3db06956bf92e50299e7fedb408f624d5e7c4156fc72a0"}
Sep 11 14:59:34.465561 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:34.465519    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56" event={"ID":"2daa6ec8-8f7b-4532-a7a5-192cd9df0793","Type":"ContainerDied","Data":"596e67e24628a72341984a5e5aa5aab469851ed8e8c30b2e385486fdcb6345ec"}
Sep 11 14:59:34.465561 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:34.465542    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="596e67e24628a72341984a5e5aa5aab469851ed8e8c30b2e385486fdcb6345ec"
Sep 11 14:59:34.468340 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:34.468317    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"
Sep 11 14:59:34.488058 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:34.488019    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-metrics-certs\") pod \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\" (UID: \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\") "
Sep 11 14:59:34.488058 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:34.488058    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-maas-api-tls\") pod \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\" (UID: \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\") "
Sep 11 14:59:34.488337 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:34.488146    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-openshift-service-ca\") pod \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\" (UID: \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\") "
Sep 11 14:59:34.488337 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:34.488188    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fp2r6\" (UniqueName: \"kubernetes.io/projected/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-kube-api-access-fp2r6\") pod \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\" (UID: \"2daa6ec8-8f7b-4532-a7a5-192cd9df0793\") "
Sep 11 14:59:34.488677 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:34.488644    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2daa6ec8-8f7b-4532-a7a5-192cd9df0793" (UID: "2daa6ec8-8f7b-4532-a7a5-192cd9df0793"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:59:34.491093 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:34.491068    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2daa6ec8-8f7b-4532-a7a5-192cd9df0793" (UID: "2daa6ec8-8f7b-4532-a7a5-192cd9df0793"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:59:34.491241 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:34.491160    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2daa6ec8-8f7b-4532-a7a5-192cd9df0793" (UID: "2daa6ec8-8f7b-4532-a7a5-192cd9df0793"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:59:34.491991 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:34.491396    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-kube-api-access-fp2r6" (OuterVolumeSpecName: "kube-api-access-fp2r6") pod "2daa6ec8-8f7b-4532-a7a5-192cd9df0793" (UID: "2daa6ec8-8f7b-4532-a7a5-192cd9df0793"). InnerVolumeSpecName "kube-api-access-fp2r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:59:34.589418 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:34.589380    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-openshift-service-ca\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:34.589418 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:34.589419    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp2r6\" (UniqueName: \"kubernetes.io/projected/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-kube-api-access-fp2r6\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:34.589616 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:34.589434    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-metrics-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:34.589616 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:34.589448    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2daa6ec8-8f7b-4532-a7a5-192cd9df0793-maas-api-tls\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:35.470887 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:35.470855    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"
Sep 11 14:59:35.488695 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:35.488663    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"]
Sep 11 14:59:35.495851 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:35.495823    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-362f7a-7456b6f67c-fzr56"]
Sep 11 14:59:36.662902 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:36.662865    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2daa6ec8-8f7b-4532-a7a5-192cd9df0793" path="/var/lib/kubelet/pods/2daa6ec8-8f7b-4532-a7a5-192cd9df0793/volumes"
Sep 11 14:59:39.470981 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:39.470951    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"
Sep 11 14:59:39.499996 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:39.499916    2568 generic.go:356] "Generic (PLEG): container finished" podID="51e90696-abd3-4ccd-b528-df0eea3b2023" containerID="bdc9afc25096dcdb57f73995eea5a007211ddac2d837e53c477bb4671bc40491" exitCode=0
Sep 11 14:59:39.500138 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:39.500023    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph" event={"ID":"51e90696-abd3-4ccd-b528-df0eea3b2023","Type":"ContainerDied","Data":"bdc9afc25096dcdb57f73995eea5a007211ddac2d837e53c477bb4671bc40491"}
Sep 11 14:59:39.500533 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:39.500507    2568 scope.go:117] "RemoveContainer" containerID="bdc9afc25096dcdb57f73995eea5a007211ddac2d837e53c477bb4671bc40491"
Sep 11 14:59:39.502051 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:39.502020    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:59:39.502139 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:39.502093    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:59:39.502139 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:39.502132    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:59:40.507110 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:40.507072    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph" event={"ID":"51e90696-abd3-4ccd-b528-df0eea3b2023","Type":"ContainerStarted","Data":"c37c4333dc76e3c7f88dedf8a449e302b8a5dd5502d0959b5b00884cbefa8db6"}
Sep 11 14:59:41.207587 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:41.207555    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:41.207762 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:41.207724    2568 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 11 14:59:41.207808 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:41.207784    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph" podUID="51e90696-abd3-4ccd-b528-df0eea3b2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 11 14:59:41.833842 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:41.832882    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"]
Sep 11 14:59:41.833842 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:41.833409    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4" podUID="52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" containerName="istio-proxy" containerID="cri-o://af00ece293f05f0ade3121c71ad983f536bbbdffaa1d0bafab1ecc7d311b5685" gracePeriod=30
Sep 11 14:59:42.207668 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:42.207589    2568 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 11 14:59:42.207668 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:42.207648    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph" podUID="51e90696-abd3-4ccd-b528-df0eea3b2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 11 14:59:43.207790 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:43.207568    2568 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 11 14:59:43.207790 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:43.207650    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph" podUID="51e90696-abd3-4ccd-b528-df0eea3b2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 11 14:59:44.207396 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:44.207357    2568 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 11 14:59:44.207634 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:44.207609    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph" podUID="51e90696-abd3-4ccd-b528-df0eea3b2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 11 14:59:45.207693 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:45.207662    2568 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 11 14:59:45.208127 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:45.207719    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph" podUID="51e90696-abd3-4ccd-b528-df0eea3b2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 11 14:59:46.207895 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:46.207856    2568 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 11 14:59:46.208271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:46.207921    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph" podUID="51e90696-abd3-4ccd-b528-df0eea3b2023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 11 14:59:47.120109 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.120043    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:59:47.211083 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.211051    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:47.211472 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.211340    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:47.212147 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.212124    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 14:59:47.224949 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.224921    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-podinfo\") pod \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") "
Sep 11 14:59:47.225065 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.224957    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-token\") pod \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") "
Sep 11 14:59:47.225065 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.224987    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istiod-ca-cert\") pod \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") "
Sep 11 14:59:47.225065 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.225014    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-workload-certs\") pod \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") "
Sep 11 14:59:47.225065 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.225047    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-envoy\") pod \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") "
Sep 11 14:59:47.225358 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.225121    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-data\") pod \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") "
Sep 11 14:59:47.225358 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.225188    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-workload-socket\") pod \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") "
Sep 11 14:59:47.225358 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.225226    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-credential-socket\") pod \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") "
Sep 11 14:59:47.225358 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.225255    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqnrn\" (UniqueName: \"kubernetes.io/projected/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-kube-api-access-hqnrn\") pod \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\" (UID: \"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c\") "
Sep 11 14:59:47.225963 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.225735    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" (UID: "52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:59:47.225963 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.225809    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-data" (OuterVolumeSpecName: "istio-data") pod "52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" (UID: "52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:59:47.225963 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.225934    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" (UID: "52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:59:47.226170 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.226075    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" (UID: "52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:59:47.226170 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.226101    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" (UID: "52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:59:47.227245 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.227226    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" (UID: "52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:59:47.227528 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.227508    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" (UID: "52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:59:47.227528 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.227519    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-token" (OuterVolumeSpecName: "istio-token") pod "52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" (UID: "52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:59:47.227723 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.227698    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-kube-api-access-hqnrn" (OuterVolumeSpecName: "kube-api-access-hqnrn") pod "52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" (UID: "52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c"). InnerVolumeSpecName "kube-api-access-hqnrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:59:47.327226 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.327191    2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-data\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:47.327226 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.327221    2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-workload-socket\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:47.327226 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.327233    2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-credential-socket\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:47.327461 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.327245    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqnrn\" (UniqueName: \"kubernetes.io/projected/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-kube-api-access-hqnrn\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:47.327461 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.327257    2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-podinfo\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:47.327461 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.327269    2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-token\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:47.327461 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.327280    2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istiod-ca-cert\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:47.327461 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.327291    2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-workload-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:47.327461 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.327302    2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c-istio-envoy\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:47.558540 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.558504    2568 generic.go:356] "Generic (PLEG): container finished" podID="52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" containerID="af00ece293f05f0ade3121c71ad983f536bbbdffaa1d0bafab1ecc7d311b5685" exitCode=0
Sep 11 14:59:47.558690 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.558556    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"
Sep 11 14:59:47.558690 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.558594    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4" event={"ID":"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c","Type":"ContainerDied","Data":"af00ece293f05f0ade3121c71ad983f536bbbdffaa1d0bafab1ecc7d311b5685"}
Sep 11 14:59:47.558690 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.558649    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4" event={"ID":"52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c","Type":"ContainerDied","Data":"c831ce82ca942ea7b03844f17ba5e6468abd718ab1054a2f1cfb9c42d56e5c4b"}
Sep 11 14:59:47.558690 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.558672    2568 scope.go:117] "RemoveContainer" containerID="af00ece293f05f0ade3121c71ad983f536bbbdffaa1d0bafab1ecc7d311b5685"
Sep 11 14:59:47.569234 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.569216    2568 scope.go:117] "RemoveContainer" containerID="d70a083c6ac54eca32a2ce9d317797669addbe3de1094d40fb2d46fc12e5a81d"
Sep 11 14:59:47.580148 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.580130    2568 scope.go:117] "RemoveContainer" containerID="af00ece293f05f0ade3121c71ad983f536bbbdffaa1d0bafab1ecc7d311b5685"
Sep 11 14:59:47.580400 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:59:47.580379    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af00ece293f05f0ade3121c71ad983f536bbbdffaa1d0bafab1ecc7d311b5685\": container with ID starting with af00ece293f05f0ade3121c71ad983f536bbbdffaa1d0bafab1ecc7d311b5685 not found: ID does not exist" containerID="af00ece293f05f0ade3121c71ad983f536bbbdffaa1d0bafab1ecc7d311b5685"
Sep 11 14:59:47.580477 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.580411    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af00ece293f05f0ade3121c71ad983f536bbbdffaa1d0bafab1ecc7d311b5685"} err="failed to get container status \"af00ece293f05f0ade3121c71ad983f536bbbdffaa1d0bafab1ecc7d311b5685\": rpc error: code = NotFound desc = could not find container \"af00ece293f05f0ade3121c71ad983f536bbbdffaa1d0bafab1ecc7d311b5685\": container with ID starting with af00ece293f05f0ade3121c71ad983f536bbbdffaa1d0bafab1ecc7d311b5685 not found: ID does not exist"
Sep 11 14:59:47.580477 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.580435    2568 scope.go:117] "RemoveContainer" containerID="d70a083c6ac54eca32a2ce9d317797669addbe3de1094d40fb2d46fc12e5a81d"
Sep 11 14:59:47.580726 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:59:47.580705    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d70a083c6ac54eca32a2ce9d317797669addbe3de1094d40fb2d46fc12e5a81d\": container with ID starting with d70a083c6ac54eca32a2ce9d317797669addbe3de1094d40fb2d46fc12e5a81d not found: ID does not exist" containerID="d70a083c6ac54eca32a2ce9d317797669addbe3de1094d40fb2d46fc12e5a81d"
Sep 11 14:59:47.580773 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.580733    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70a083c6ac54eca32a2ce9d317797669addbe3de1094d40fb2d46fc12e5a81d"} err="failed to get container status \"d70a083c6ac54eca32a2ce9d317797669addbe3de1094d40fb2d46fc12e5a81d\": rpc error: code = NotFound desc = could not find container \"d70a083c6ac54eca32a2ce9d317797669addbe3de1094d40fb2d46fc12e5a81d\": container with ID starting with d70a083c6ac54eca32a2ce9d317797669addbe3de1094d40fb2d46fc12e5a81d not found: ID does not exist"
Sep 11 14:59:47.582073 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.582054    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"]
Sep 11 14:59:47.585239 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:47.585220    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-362f7a-openshift-default-6ff595f56-ft5p4"]
Sep 11 14:59:48.661806 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:48.661764    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" path="/var/lib/kubelet/pods/52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c/volumes"
Sep 11 14:59:50.664024 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.663994    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"]
Sep 11 14:59:50.664390 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.664384    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2daa6ec8-8f7b-4532-a7a5-192cd9df0793" containerName="maas-api"
Sep 11 14:59:50.664430 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.664394    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2daa6ec8-8f7b-4532-a7a5-192cd9df0793" containerName="maas-api"
Sep 11 14:59:50.664430 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.664412    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" containerName="istio-proxy"
Sep 11 14:59:50.664430 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.664418    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" containerName="istio-proxy"
Sep 11 14:59:50.664430 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.664428    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" containerName="istio-proxy"
Sep 11 14:59:50.664552 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.664433    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" containerName="istio-proxy"
Sep 11 14:59:50.664552 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.664500    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2daa6ec8-8f7b-4532-a7a5-192cd9df0793" containerName="maas-api"
Sep 11 14:59:50.664552 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.664508    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" containerName="istio-proxy"
Sep 11 14:59:50.664552 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.664515    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="52f6cfcf-64db-4dfb-ae3b-bc8bc25fbf5c" containerName="istio-proxy"
Sep 11 14:59:50.667604 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.667588    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 14:59:50.670338 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.670313    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-837632\"/\"openshift-service-ca.crt\""
Sep 11 14:59:50.670574 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.670557    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-837632\"/\"kube-root-ca.crt\""
Sep 11 14:59:50.671718 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.671700    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-837632\"/\"e2e-conflict-b-372348b4-kserve-self-signed-certs\""
Sep 11 14:59:50.671778 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.671761    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-837632\"/\"default-dockercfg-cz84q\""
Sep 11 14:59:50.677382 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.677359    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"]
Sep 11 14:59:50.760378 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.760342    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxdg5\" (UniqueName: \"kubernetes.io/projected/eb779e25-14e3-4e2a-9526-52fe2882bd13-kube-api-access-pxdg5\") pod \"e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 14:59:50.760378 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.760379    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-model-cache\") pod \"e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 14:59:50.760618 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.760434    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb779e25-14e3-4e2a-9526-52fe2882bd13-tls-certs\") pod \"e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 14:59:50.760618 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.760453    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-home\") pod \"e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 14:59:50.760618 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.760575    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-tmp-dir\") pod \"e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 14:59:50.760730 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.760630    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-dshm\") pod \"e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 14:59:50.861788 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.861752    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-tmp-dir\") pod \"e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 14:59:50.861982 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.861801    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-dshm\") pod \"e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 14:59:50.861982 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.861850    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxdg5\" (UniqueName: \"kubernetes.io/projected/eb779e25-14e3-4e2a-9526-52fe2882bd13-kube-api-access-pxdg5\") pod \"e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 14:59:50.861982 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.861880    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-model-cache\") pod \"e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 14:59:50.861982 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.861941    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb779e25-14e3-4e2a-9526-52fe2882bd13-tls-certs\") pod \"e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 14:59:50.861982 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.861965    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-home\") pod \"e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 14:59:50.862261 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.862240    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-tmp-dir\") pod \"e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 14:59:50.862308 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.862291    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-model-cache\") pod \"e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 14:59:50.862406 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.862386    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-home\") pod \"e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 14:59:50.864195 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.864167    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-dshm\") pod \"e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 14:59:50.864346 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.864330    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb779e25-14e3-4e2a-9526-52fe2882bd13-tls-certs\") pod \"e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 14:59:50.870331 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.870313    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxdg5\" (UniqueName: \"kubernetes.io/projected/eb779e25-14e3-4e2a-9526-52fe2882bd13-kube-api-access-pxdg5\") pod \"e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") " pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 14:59:50.977856 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:50.977759    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 14:59:51.254313 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:51.254282    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"]
Sep 11 14:59:51.256582 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:59:51.256557    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb779e25_14e3_4e2a_9526_52fe2882bd13.slice/crio-9b91b38f4a150ab98e8952a68631670153420606fa6ddc31cd05ac5c815139c0 WatchSource:0}: Error finding container 9b91b38f4a150ab98e8952a68631670153420606fa6ddc31cd05ac5c815139c0: Status 404 returned error can't find the container with id 9b91b38f4a150ab98e8952a68631670153420606fa6ddc31cd05ac5c815139c0
Sep 11 14:59:51.582243 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:51.582205    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q" event={"ID":"eb779e25-14e3-4e2a-9526-52fe2882bd13","Type":"ContainerStarted","Data":"d4fc1b85321aef34738d76d71fe6ce04de25f050c710dfa9750c9865424ceb86"}
Sep 11 14:59:51.582399 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:51.582252    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q" event={"ID":"eb779e25-14e3-4e2a-9526-52fe2882bd13","Type":"ContainerStarted","Data":"9b91b38f4a150ab98e8952a68631670153420606fa6ddc31cd05ac5c815139c0"}
Sep 11 14:59:51.602480 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:51.602427    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q" podStartSLOduration=1.60241071 podStartE2EDuration="1.60241071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:59:51.601097895 +0000 UTC m=+1171.389362193" watchObservedRunningTime="2026-09-11 14:59:51.60241071 +0000 UTC m=+1171.390675008"
Sep 11 14:59:53.850244 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:53.850213    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v"]
Sep 11 14:59:53.850687 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:53.850426    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v" podUID="2341a23b-500e-4832-abf6-b3c858e84e0a" containerName="payload-processing" containerID="cri-o://3fa7d21939e6d3f1e3f6bc899435edcfe2f3c70840bc3980e4a1061cb704fb7a" gracePeriod=30
Sep 11 14:59:53.856005 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:53.855893    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk"]
Sep 11 14:59:53.856161 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:53.856095    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk" podUID="c560d780-16cf-4739-851a-d034eeb090bb" containerName="payload-pre-processing" containerID="cri-o://e8a183489291889a9c43de7d9e8b3e61b253505703a744968eaeb07c50d71c5d" gracePeriod=30
Sep 11 14:59:54.139392 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.139189    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk"
Sep 11 14:59:54.176081 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.176052    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v"
Sep 11 14:59:54.196162 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.196127    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c560d780-16cf-4739-851a-d034eeb090bb-plugins-config-volume\") pod \"c560d780-16cf-4739-851a-d034eeb090bb\" (UID: \"c560d780-16cf-4739-851a-d034eeb090bb\") "
Sep 11 14:59:54.196359 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.196335    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2b65\" (UniqueName: \"kubernetes.io/projected/c560d780-16cf-4739-851a-d034eeb090bb-kube-api-access-v2b65\") pod \"c560d780-16cf-4739-851a-d034eeb090bb\" (UID: \"c560d780-16cf-4739-851a-d034eeb090bb\") "
Sep 11 14:59:54.196673 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.196640    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c560d780-16cf-4739-851a-d034eeb090bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c560d780-16cf-4739-851a-d034eeb090bb" (UID: "c560d780-16cf-4739-851a-d034eeb090bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:59:54.198577 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.198551    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c560d780-16cf-4739-851a-d034eeb090bb-kube-api-access-v2b65" (OuterVolumeSpecName: "kube-api-access-v2b65") pod "c560d780-16cf-4739-851a-d034eeb090bb" (UID: "c560d780-16cf-4739-851a-d034eeb090bb"). InnerVolumeSpecName "kube-api-access-v2b65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:59:54.297349 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.297304    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2341a23b-500e-4832-abf6-b3c858e84e0a-plugins-config-volume\") pod \"2341a23b-500e-4832-abf6-b3c858e84e0a\" (UID: \"2341a23b-500e-4832-abf6-b3c858e84e0a\") "
Sep 11 14:59:54.297531 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.297402    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hdd2\" (UniqueName: \"kubernetes.io/projected/2341a23b-500e-4832-abf6-b3c858e84e0a-kube-api-access-2hdd2\") pod \"2341a23b-500e-4832-abf6-b3c858e84e0a\" (UID: \"2341a23b-500e-4832-abf6-b3c858e84e0a\") "
Sep 11 14:59:54.297611 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.297600    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c560d780-16cf-4739-851a-d034eeb090bb-plugins-config-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:54.297646 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.297612    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2b65\" (UniqueName: \"kubernetes.io/projected/c560d780-16cf-4739-851a-d034eeb090bb-kube-api-access-v2b65\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:54.297684 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.297656    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2341a23b-500e-4832-abf6-b3c858e84e0a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2341a23b-500e-4832-abf6-b3c858e84e0a" (UID: "2341a23b-500e-4832-abf6-b3c858e84e0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:59:54.299386 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.299367    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2341a23b-500e-4832-abf6-b3c858e84e0a-kube-api-access-2hdd2" (OuterVolumeSpecName: "kube-api-access-2hdd2") pod "2341a23b-500e-4832-abf6-b3c858e84e0a" (UID: "2341a23b-500e-4832-abf6-b3c858e84e0a"). InnerVolumeSpecName "kube-api-access-2hdd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:59:54.399215 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.399136    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hdd2\" (UniqueName: \"kubernetes.io/projected/2341a23b-500e-4832-abf6-b3c858e84e0a-kube-api-access-2hdd2\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:54.399215 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.399164    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2341a23b-500e-4832-abf6-b3c858e84e0a-plugins-config-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 14:59:54.599890 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.599854    2568 generic.go:356] "Generic (PLEG): container finished" podID="c560d780-16cf-4739-851a-d034eeb090bb" containerID="e8a183489291889a9c43de7d9e8b3e61b253505703a744968eaeb07c50d71c5d" exitCode=0
Sep 11 14:59:54.600172 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.599918    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk"
Sep 11 14:59:54.600172 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.599938    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk" event={"ID":"c560d780-16cf-4739-851a-d034eeb090bb","Type":"ContainerDied","Data":"e8a183489291889a9c43de7d9e8b3e61b253505703a744968eaeb07c50d71c5d"}
Sep 11 14:59:54.600172 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.600026    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk" event={"ID":"c560d780-16cf-4739-851a-d034eeb090bb","Type":"ContainerDied","Data":"72cfbec9151fb61facd49963cd4457edad601132e7656d54cfe68754f456fa32"}
Sep 11 14:59:54.600172 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.600060    2568 scope.go:117] "RemoveContainer" containerID="e8a183489291889a9c43de7d9e8b3e61b253505703a744968eaeb07c50d71c5d"
Sep 11 14:59:54.601338 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.601311    2568 generic.go:356] "Generic (PLEG): container finished" podID="2341a23b-500e-4832-abf6-b3c858e84e0a" containerID="3fa7d21939e6d3f1e3f6bc899435edcfe2f3c70840bc3980e4a1061cb704fb7a" exitCode=0
Sep 11 14:59:54.601446 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.601355    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v" event={"ID":"2341a23b-500e-4832-abf6-b3c858e84e0a","Type":"ContainerDied","Data":"3fa7d21939e6d3f1e3f6bc899435edcfe2f3c70840bc3980e4a1061cb704fb7a"}
Sep 11 14:59:54.601446 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.601360    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v"
Sep 11 14:59:54.601446 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.601390    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v" event={"ID":"2341a23b-500e-4832-abf6-b3c858e84e0a","Type":"ContainerDied","Data":"ef629f572752fc762dfff8663d0e0c9e639f6e751492c7c49d928a52e0410cf2"}
Sep 11 14:59:54.609804 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.609619    2568 scope.go:117] "RemoveContainer" containerID="e8a183489291889a9c43de7d9e8b3e61b253505703a744968eaeb07c50d71c5d"
Sep 11 14:59:54.610020 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:59:54.609989    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a183489291889a9c43de7d9e8b3e61b253505703a744968eaeb07c50d71c5d\": container with ID starting with e8a183489291889a9c43de7d9e8b3e61b253505703a744968eaeb07c50d71c5d not found: ID does not exist" containerID="e8a183489291889a9c43de7d9e8b3e61b253505703a744968eaeb07c50d71c5d"
Sep 11 14:59:54.610133 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.610030    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a183489291889a9c43de7d9e8b3e61b253505703a744968eaeb07c50d71c5d"} err="failed to get container status \"e8a183489291889a9c43de7d9e8b3e61b253505703a744968eaeb07c50d71c5d\": rpc error: code = NotFound desc = could not find container \"e8a183489291889a9c43de7d9e8b3e61b253505703a744968eaeb07c50d71c5d\": container with ID starting with e8a183489291889a9c43de7d9e8b3e61b253505703a744968eaeb07c50d71c5d not found: ID does not exist"
Sep 11 14:59:54.610133 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.610055    2568 scope.go:117] "RemoveContainer" containerID="3fa7d21939e6d3f1e3f6bc899435edcfe2f3c70840bc3980e4a1061cb704fb7a"
Sep 11 14:59:54.618990 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.618972    2568 scope.go:117] "RemoveContainer" containerID="3fa7d21939e6d3f1e3f6bc899435edcfe2f3c70840bc3980e4a1061cb704fb7a"
Sep 11 14:59:54.619282 ip-10-0-133-126 kubenswrapper[2568]: E0911 14:59:54.619263    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa7d21939e6d3f1e3f6bc899435edcfe2f3c70840bc3980e4a1061cb704fb7a\": container with ID starting with 3fa7d21939e6d3f1e3f6bc899435edcfe2f3c70840bc3980e4a1061cb704fb7a not found: ID does not exist" containerID="3fa7d21939e6d3f1e3f6bc899435edcfe2f3c70840bc3980e4a1061cb704fb7a"
Sep 11 14:59:54.619321 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.619291    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa7d21939e6d3f1e3f6bc899435edcfe2f3c70840bc3980e4a1061cb704fb7a"} err="failed to get container status \"3fa7d21939e6d3f1e3f6bc899435edcfe2f3c70840bc3980e4a1061cb704fb7a\": rpc error: code = NotFound desc = could not find container \"3fa7d21939e6d3f1e3f6bc899435edcfe2f3c70840bc3980e4a1061cb704fb7a\": container with ID starting with 3fa7d21939e6d3f1e3f6bc899435edcfe2f3c70840bc3980e4a1061cb704fb7a not found: ID does not exist"
Sep 11 14:59:54.625221 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.625200    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk"]
Sep 11 14:59:54.631360 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.631337    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-90d6f5-6869fb7566-qw7rk"]
Sep 11 14:59:54.646777 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.646750    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v"]
Sep 11 14:59:54.655708 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.655635    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-90d6f5-6cb8d447d7-hm69v"]
Sep 11 14:59:54.663195 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.663140    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2341a23b-500e-4832-abf6-b3c858e84e0a" path="/var/lib/kubelet/pods/2341a23b-500e-4832-abf6-b3c858e84e0a/volumes"
Sep 11 14:59:54.663715 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.663692    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c560d780-16cf-4739-851a-d034eeb090bb" path="/var/lib/kubelet/pods/c560d780-16cf-4739-851a-d034eeb090bb/volumes"
Sep 11 14:59:54.665788 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.665760    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"]
Sep 11 14:59:54.666196 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.666182    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2341a23b-500e-4832-abf6-b3c858e84e0a" containerName="payload-processing"
Sep 11 14:59:54.666196 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.666197    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2341a23b-500e-4832-abf6-b3c858e84e0a" containerName="payload-processing"
Sep 11 14:59:54.666305 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.666210    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c560d780-16cf-4739-851a-d034eeb090bb" containerName="payload-pre-processing"
Sep 11 14:59:54.666305 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.666215    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c560d780-16cf-4739-851a-d034eeb090bb" containerName="payload-pre-processing"
Sep 11 14:59:54.666305 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.666278    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c560d780-16cf-4739-851a-d034eeb090bb" containerName="payload-pre-processing"
Sep 11 14:59:54.666305 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.666287    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2341a23b-500e-4832-abf6-b3c858e84e0a" containerName="payload-processing"
Sep 11 14:59:54.670741 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.670724    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.673447 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.673430    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a843ae03-openshift-default-dockercfg-fbdjf\""
Sep 11 14:59:54.684390 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.684368    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"]
Sep 11 14:59:54.803430 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.803391    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw98q\" (UniqueName: \"kubernetes.io/projected/c5afaa36-a4f1-4239-9e4d-a071ba934f99-kube-api-access-lw98q\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.803633 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.803453    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-credential-socket\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.803633 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.803520    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-envoy\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.803633 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.803600    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-data\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.803806 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.803645    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-workload-certs\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.803806 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.803672    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-token\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.803806 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.803693    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-podinfo\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.803986 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.803855    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-workload-socket\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.803986 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.803887    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istiod-ca-cert\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.905346 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.905310    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istiod-ca-cert\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.905346 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.905349    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw98q\" (UniqueName: \"kubernetes.io/projected/c5afaa36-a4f1-4239-9e4d-a071ba934f99-kube-api-access-lw98q\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.906520 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.905394    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-credential-socket\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.906520 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.905414    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-envoy\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.906520 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.905483    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-data\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.906520 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.905530    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-workload-certs\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.906520 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.905568    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-token\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.906520 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.905599    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-podinfo\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.906520 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.905640    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-workload-socket\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.906520 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.905908    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-data\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.906520 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.905987    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-workload-certs\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.906520 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.906013    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-workload-socket\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.906520 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.906141    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-credential-socket\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.906520 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.906198    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istiod-ca-cert\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.907869 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.907846    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-envoy\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.908199 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.908178    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-podinfo\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.915375 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.915347    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw98q\" (UniqueName: \"kubernetes.io/projected/c5afaa36-a4f1-4239-9e4d-a071ba934f99-kube-api-access-lw98q\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.915508 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.915359    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-token\") pod \"e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") " pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:54.984887 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:54.984855    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:55.268429 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:55.268400    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"]
Sep 11 14:59:55.270255 ip-10-0-133-126 kubenswrapper[2568]: W0911 14:59:55.270218    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5afaa36_a4f1_4239_9e4d_a071ba934f99.slice/crio-a13cc65e3f23a52d9c791b93ac418477b5748c68c02e2792161e77ffbe5a2cdd WatchSource:0}: Error finding container a13cc65e3f23a52d9c791b93ac418477b5748c68c02e2792161e77ffbe5a2cdd: Status 404 returned error can't find the container with id a13cc65e3f23a52d9c791b93ac418477b5748c68c02e2792161e77ffbe5a2cdd
Sep 11 14:59:55.272659 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:55.272623    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:59:55.272786 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:55.272708    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:59:55.272786 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:55.272741    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 14:59:55.610438 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:55.610404    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9" event={"ID":"c5afaa36-a4f1-4239-9e4d-a071ba934f99","Type":"ContainerStarted","Data":"e51541494a1ccf6b0758638122e87b687ffbd1b3f4a0827f2eabd1c7608744b7"}
Sep 11 14:59:55.610602 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:55.610455    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9" event={"ID":"c5afaa36-a4f1-4239-9e4d-a071ba934f99","Type":"ContainerStarted","Data":"a13cc65e3f23a52d9c791b93ac418477b5748c68c02e2792161e77ffbe5a2cdd"}
Sep 11 14:59:55.659531 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:55.659479    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9" podStartSLOduration=1.659464848 podStartE2EDuration="1.659464848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:59:55.656596963 +0000 UTC m=+1175.444861263" watchObservedRunningTime="2026-09-11 14:59:55.659464848 +0000 UTC m=+1175.447729146"
Sep 11 14:59:55.985276 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:55.985189    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:56.990655 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:56.990623    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:57.622749 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:57.622698    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:57.623758 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:57.623730    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 14:59:59.585530 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.585493    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp"]
Sep 11 14:59:59.589386 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.589369    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp"
Sep 11 14:59:59.591969 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.591947    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a843ae03\""
Sep 11 14:59:59.592077 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.591947    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a843ae03-dockercfg-zg42s\""
Sep 11 14:59:59.600079 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.600053    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp"]
Sep 11 14:59:59.618207 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.618171    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h"]
Sep 11 14:59:59.621900 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.621879    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h"
Sep 11 14:59:59.630703 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.630675    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h"]
Sep 11 14:59:59.760942 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.760900    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f46167db-ae60-4728-a579-e69655ced01e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp\" (UID: \"f46167db-ae60-4728-a579-e69655ced01e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp"
Sep 11 14:59:59.761138 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.761076    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbj22\" (UniqueName: \"kubernetes.io/projected/f46167db-ae60-4728-a579-e69655ced01e-kube-api-access-fbj22\") pod \"payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp\" (UID: \"f46167db-ae60-4728-a579-e69655ced01e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp"
Sep 11 14:59:59.761197 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.761153    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pftm\" (UniqueName: \"kubernetes.io/projected/ce2d526b-0807-4b87-a14c-1f3ac57d14a4-kube-api-access-4pftm\") pod \"payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h\" (UID: \"ce2d526b-0807-4b87-a14c-1f3ac57d14a4\") " pod="openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h"
Sep 11 14:59:59.761271 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.761242    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce2d526b-0807-4b87-a14c-1f3ac57d14a4-plugins-config-volume\") pod \"payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h\" (UID: \"ce2d526b-0807-4b87-a14c-1f3ac57d14a4\") " pod="openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h"
Sep 11 14:59:59.862345 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.862252    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f46167db-ae60-4728-a579-e69655ced01e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp\" (UID: \"f46167db-ae60-4728-a579-e69655ced01e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp"
Sep 11 14:59:59.862529 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.862360    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbj22\" (UniqueName: \"kubernetes.io/projected/f46167db-ae60-4728-a579-e69655ced01e-kube-api-access-fbj22\") pod \"payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp\" (UID: \"f46167db-ae60-4728-a579-e69655ced01e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp"
Sep 11 14:59:59.862529 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.862391    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pftm\" (UniqueName: \"kubernetes.io/projected/ce2d526b-0807-4b87-a14c-1f3ac57d14a4-kube-api-access-4pftm\") pod \"payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h\" (UID: \"ce2d526b-0807-4b87-a14c-1f3ac57d14a4\") " pod="openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h"
Sep 11 14:59:59.862529 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.862429    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce2d526b-0807-4b87-a14c-1f3ac57d14a4-plugins-config-volume\") pod \"payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h\" (UID: \"ce2d526b-0807-4b87-a14c-1f3ac57d14a4\") " pod="openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h"
Sep 11 14:59:59.863241 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.863217    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce2d526b-0807-4b87-a14c-1f3ac57d14a4-plugins-config-volume\") pod \"payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h\" (UID: \"ce2d526b-0807-4b87-a14c-1f3ac57d14a4\") " pod="openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h"
Sep 11 14:59:59.863291 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.863228    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f46167db-ae60-4728-a579-e69655ced01e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp\" (UID: \"f46167db-ae60-4728-a579-e69655ced01e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp"
Sep 11 14:59:59.871115 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.871086    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbj22\" (UniqueName: \"kubernetes.io/projected/f46167db-ae60-4728-a579-e69655ced01e-kube-api-access-fbj22\") pod \"payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp\" (UID: \"f46167db-ae60-4728-a579-e69655ced01e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp"
Sep 11 14:59:59.871115 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.871096    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pftm\" (UniqueName: \"kubernetes.io/projected/ce2d526b-0807-4b87-a14c-1f3ac57d14a4-kube-api-access-4pftm\") pod \"payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h\" (UID: \"ce2d526b-0807-4b87-a14c-1f3ac57d14a4\") " pod="openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h"
Sep 11 14:59:59.901077 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.901045    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp"
Sep 11 14:59:59.934153 ip-10-0-133-126 kubenswrapper[2568]: I0911 14:59:59.934123    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h"
Sep 11 15:00:00.178153 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:00.178114    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp"]
Sep 11 15:00:00.179462 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:00:00.179440    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46167db_ae60_4728_a579_e69655ced01e.slice/crio-785300fae4fdf2a319479db13733ef086cda9844ca9d1905257421718af27da9 WatchSource:0}: Error finding container 785300fae4fdf2a319479db13733ef086cda9844ca9d1905257421718af27da9: Status 404 returned error can't find the container with id 785300fae4fdf2a319479db13733ef086cda9844ca9d1905257421718af27da9
Sep 11 15:00:00.230710 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:00.230683    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h"]
Sep 11 15:00:00.231599 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:00:00.231575    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2d526b_0807_4b87_a14c_1f3ac57d14a4.slice/crio-4d5623ad2bc0d7e3707ea15a31a5333a8bcc6fe4d153c1bfd3a8ad2455251a64 WatchSource:0}: Error finding container 4d5623ad2bc0d7e3707ea15a31a5333a8bcc6fe4d153c1bfd3a8ad2455251a64: Status 404 returned error can't find the container with id 4d5623ad2bc0d7e3707ea15a31a5333a8bcc6fe4d153c1bfd3a8ad2455251a64
Sep 11 15:00:00.649198 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:00.649159    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp" event={"ID":"f46167db-ae60-4728-a579-e69655ced01e","Type":"ContainerStarted","Data":"e840b5b0474dadbe502951626a6147d926551428157241df8e761a400d3fdc6b"}
Sep 11 15:00:00.649745 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:00.649235    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp" event={"ID":"f46167db-ae60-4728-a579-e69655ced01e","Type":"ContainerStarted","Data":"785300fae4fdf2a319479db13733ef086cda9844ca9d1905257421718af27da9"}
Sep 11 15:00:00.649745 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:00.649383    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp"
Sep 11 15:00:00.651128 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:00.651099    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h" event={"ID":"ce2d526b-0807-4b87-a14c-1f3ac57d14a4","Type":"ContainerStarted","Data":"d02300f3ba689c39be68c891e2a8171a0f364e53eec4f1436d49f55f7eb2d328"}
Sep 11 15:00:00.651264 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:00.651145    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h" event={"ID":"ce2d526b-0807-4b87-a14c-1f3ac57d14a4","Type":"ContainerStarted","Data":"4d5623ad2bc0d7e3707ea15a31a5333a8bcc6fe4d153c1bfd3a8ad2455251a64"}
Sep 11 15:00:00.651264 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:00.651173    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h"
Sep 11 15:00:00.669573 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:00.669517    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp" podStartSLOduration=1.6694995019999999 podStartE2EDuration="1.669499502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:00:00.666343501 +0000 UTC m=+1180.454607800" watchObservedRunningTime="2026-09-11 15:00:00.669499502 +0000 UTC m=+1180.457763807"
Sep 11 15:00:00.692587 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:00.692518    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h" podStartSLOduration=1.692499016 podStartE2EDuration="1.692499016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:00:00.690322524 +0000 UTC m=+1180.478586825" watchObservedRunningTime="2026-09-11 15:00:00.692499016 +0000 UTC m=+1180.480763334"
Sep 11 15:00:00.978769 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:00.978680    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 15:00:00.978769 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:00.978736    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 15:00:00.991804 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:00.991770    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 15:00:01.668461 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:01.668432    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 15:00:11.659958 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:11.659923    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp"
Sep 11 15:00:11.660396 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:11.660334    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h"
Sep 11 15:00:20.711353 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:20.711317    2568 scope.go:117] "RemoveContainer" containerID="bb5ff971effee8ad3a3db06956bf92e50299e7fedb408f624d5e7c4156fc72a0"
Sep 11 15:00:28.664971 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:28.664934    2568 patch_prober.go:28] interesting pod/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 15:00:28.665459 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:28.665008    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9" podUID="c5afaa36-a4f1-4239-9e4d-a071ba934f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 15:00:30.831355 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:30.831322    2568 generic.go:356] "Generic (PLEG): container finished" podID="c5afaa36-a4f1-4239-9e4d-a071ba934f99" containerID="e51541494a1ccf6b0758638122e87b687ffbd1b3f4a0827f2eabd1c7608744b7" exitCode=0
Sep 11 15:00:30.831772 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:30.831405    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9" event={"ID":"c5afaa36-a4f1-4239-9e4d-a071ba934f99","Type":"ContainerDied","Data":"e51541494a1ccf6b0758638122e87b687ffbd1b3f4a0827f2eabd1c7608744b7"}
Sep 11 15:00:30.831851 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:30.831836    2568 scope.go:117] "RemoveContainer" containerID="e51541494a1ccf6b0758638122e87b687ffbd1b3f4a0827f2eabd1c7608744b7"
Sep 11 15:00:30.833022 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:30.832997    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:00:30.833090 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:30.833062    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:00:30.833134 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:30.833096    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:00:31.838275 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:31.838239    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9" event={"ID":"c5afaa36-a4f1-4239-9e4d-a071ba934f99","Type":"ContainerStarted","Data":"23e9c1bde7c407527d40453b8d380c5e791ac6a8b012cd74ee36dbf65d4a4b5c"}
Sep 11 15:00:31.985542 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:31.985509    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 15:00:31.985715 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:31.985693    2568 patch_prober.go:28] interesting pod/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 11 15:00:31.985772 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:31.985750    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9" podUID="c5afaa36-a4f1-4239-9e4d-a071ba934f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 11 15:00:32.985560 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:32.985520    2568 patch_prober.go:28] interesting pod/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 11 15:00:32.985972 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:32.985585    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9" podUID="c5afaa36-a4f1-4239-9e4d-a071ba934f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 11 15:00:33.985870 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:33.985807    2568 patch_prober.go:28] interesting pod/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 11 15:00:33.986266 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:33.985893    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9" podUID="c5afaa36-a4f1-4239-9e4d-a071ba934f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 11 15:00:34.985592 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:34.985548    2568 patch_prober.go:28] interesting pod/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 11 15:00:34.985811 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:34.985632    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9" podUID="c5afaa36-a4f1-4239-9e4d-a071ba934f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 11 15:00:35.986264 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:35.986222    2568 patch_prober.go:28] interesting pod/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 11 15:00:35.986742 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:35.986309    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9" podUID="c5afaa36-a4f1-4239-9e4d-a071ba934f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 11 15:00:36.985844 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:36.985796    2568 patch_prober.go:28] interesting pod/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 11 15:00:36.986020 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:36.985871    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9" podUID="c5afaa36-a4f1-4239-9e4d-a071ba934f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 11 15:00:37.989883 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:37.989851    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 15:00:37.990267 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:37.990086    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 15:00:37.990897 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:37.990878    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 15:00:46.036432 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:46.036396    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2"]
Sep 11 15:00:46.041751 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:46.041726    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2"
Sep 11 15:00:46.044854 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:46.044810    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-48898f-dockercfg-5vgzg\""
Sep 11 15:00:46.045064 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:46.045040    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-48898f\""
Sep 11 15:00:46.050484 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:46.050457    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2"]
Sep 11 15:00:46.139389 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:46.139339    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cszg\" (UniqueName: \"kubernetes.io/projected/40f2dbf4-4caf-4623-adcc-9079eb94ed7a-kube-api-access-9cszg\") pod \"payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2\" (UID: \"40f2dbf4-4caf-4623-adcc-9079eb94ed7a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2"
Sep 11 15:00:46.139588 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:46.139395    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f2dbf4-4caf-4623-adcc-9079eb94ed7a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2\" (UID: \"40f2dbf4-4caf-4623-adcc-9079eb94ed7a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2"
Sep 11 15:00:46.240214 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:46.240178    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cszg\" (UniqueName: \"kubernetes.io/projected/40f2dbf4-4caf-4623-adcc-9079eb94ed7a-kube-api-access-9cszg\") pod \"payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2\" (UID: \"40f2dbf4-4caf-4623-adcc-9079eb94ed7a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2"
Sep 11 15:00:46.240387 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:46.240220    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f2dbf4-4caf-4623-adcc-9079eb94ed7a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2\" (UID: \"40f2dbf4-4caf-4623-adcc-9079eb94ed7a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2"
Sep 11 15:00:46.241005 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:46.240985    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f2dbf4-4caf-4623-adcc-9079eb94ed7a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2\" (UID: \"40f2dbf4-4caf-4623-adcc-9079eb94ed7a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2"
Sep 11 15:00:46.248802 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:46.248775    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cszg\" (UniqueName: \"kubernetes.io/projected/40f2dbf4-4caf-4623-adcc-9079eb94ed7a-kube-api-access-9cszg\") pod \"payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2\" (UID: \"40f2dbf4-4caf-4623-adcc-9079eb94ed7a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2"
Sep 11 15:00:46.353984 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:46.353902    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2"
Sep 11 15:00:46.632668 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:46.632642    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2"]
Sep 11 15:00:46.634272 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:00:46.634245    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f2dbf4_4caf_4623_adcc_9079eb94ed7a.slice/crio-d23b4cebc986d5a70dcd5d4fcb6924d3c619f12a0ed248194b73f26d94e1566e WatchSource:0}: Error finding container d23b4cebc986d5a70dcd5d4fcb6924d3c619f12a0ed248194b73f26d94e1566e: Status 404 returned error can't find the container with id d23b4cebc986d5a70dcd5d4fcb6924d3c619f12a0ed248194b73f26d94e1566e
Sep 11 15:00:46.925729 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:46.925644    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2" event={"ID":"40f2dbf4-4caf-4623-adcc-9079eb94ed7a","Type":"ContainerStarted","Data":"3961c68ccc92499ce3bca21fcfa8ad5c4a44e9286b80b880ed87abe9eb0c9014"}
Sep 11 15:00:46.925729 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:46.925697    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2" event={"ID":"40f2dbf4-4caf-4623-adcc-9079eb94ed7a","Type":"ContainerStarted","Data":"d23b4cebc986d5a70dcd5d4fcb6924d3c619f12a0ed248194b73f26d94e1566e"}
Sep 11 15:00:46.925931 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:46.925803    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2"
Sep 11 15:00:46.944454 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:46.944398    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2" podStartSLOduration=0.944379599 podStartE2EDuration="944.379599ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:00:46.943882722 +0000 UTC m=+1226.732147022" watchObservedRunningTime="2026-09-11 15:00:46.944379599 +0000 UTC m=+1226.732643899"
Sep 11 15:00:57.934194 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:00:57.934164    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2"
Sep 11 15:01:20.471778 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:20.471659    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"]
Sep 11 15:01:20.475798 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:20.475776    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"
Sep 11 15:01:20.477110 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:20.477086    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baaaba03-d432-4125-9978-7d63d9f7e5ca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls\" (UID: \"baaaba03-d432-4125-9978-7d63d9f7e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"
Sep 11 15:01:20.477284 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:20.477266    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baaaba03-d432-4125-9978-7d63d9f7e5ca-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls\" (UID: \"baaaba03-d432-4125-9978-7d63d9f7e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"
Sep 11 15:01:20.477424 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:20.477408    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbwwj\" (UniqueName: \"kubernetes.io/projected/baaaba03-d432-4125-9978-7d63d9f7e5ca-kube-api-access-sbwwj\") pod \"maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls\" (UID: \"baaaba03-d432-4125-9978-7d63d9f7e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"
Sep 11 15:01:20.477750 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:20.477730    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baaaba03-d432-4125-9978-7d63d9f7e5ca-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls\" (UID: \"baaaba03-d432-4125-9978-7d63d9f7e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"
Sep 11 15:01:20.478877 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:20.478855    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-928943-serving-cert\""
Sep 11 15:01:20.488663 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:20.488639    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"]
Sep 11 15:01:20.578987 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:20.578942    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baaaba03-d432-4125-9978-7d63d9f7e5ca-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls\" (UID: \"baaaba03-d432-4125-9978-7d63d9f7e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"
Sep 11 15:01:20.579193 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:20.579018    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baaaba03-d432-4125-9978-7d63d9f7e5ca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls\" (UID: \"baaaba03-d432-4125-9978-7d63d9f7e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"
Sep 11 15:01:20.579193 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:20.579046    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baaaba03-d432-4125-9978-7d63d9f7e5ca-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls\" (UID: \"baaaba03-d432-4125-9978-7d63d9f7e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"
Sep 11 15:01:20.579193 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:20.579072    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbwwj\" (UniqueName: \"kubernetes.io/projected/baaaba03-d432-4125-9978-7d63d9f7e5ca-kube-api-access-sbwwj\") pod \"maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls\" (UID: \"baaaba03-d432-4125-9978-7d63d9f7e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"
Sep 11 15:01:20.579797 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:20.579758    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baaaba03-d432-4125-9978-7d63d9f7e5ca-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls\" (UID: \"baaaba03-d432-4125-9978-7d63d9f7e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"
Sep 11 15:01:20.583985 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:20.583073    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-928943-serving-cert\""
Sep 11 15:01:20.583985 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:20.583848    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baaaba03-d432-4125-9978-7d63d9f7e5ca-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls\" (UID: \"baaaba03-d432-4125-9978-7d63d9f7e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"
Sep 11 15:01:20.592044 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:20.589144    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbwwj\" (UniqueName: \"kubernetes.io/projected/baaaba03-d432-4125-9978-7d63d9f7e5ca-kube-api-access-sbwwj\") pod \"maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls\" (UID: \"baaaba03-d432-4125-9978-7d63d9f7e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"
Sep 11 15:01:20.592044 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:01:20.589225    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-serving-cert: secret "maas-api-e2e-shared-b-w3-928943-serving-cert" not found
Sep 11 15:01:20.592044 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:01:20.589319    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baaaba03-d432-4125-9978-7d63d9f7e5ca-maas-api-tls podName:baaaba03-d432-4125-9978-7d63d9f7e5ca nodeName:}" failed. No retries permitted until 2026-09-11 15:01:21.089297828 +0000 UTC m=+1260.877562118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/baaaba03-d432-4125-9978-7d63d9f7e5ca-maas-api-tls") pod "maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls" (UID: "baaaba03-d432-4125-9978-7d63d9f7e5ca") : secret "maas-api-e2e-shared-b-w3-928943-serving-cert" not found
Sep 11 15:01:21.186212 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:21.186173    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baaaba03-d432-4125-9978-7d63d9f7e5ca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls\" (UID: \"baaaba03-d432-4125-9978-7d63d9f7e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"
Sep 11 15:01:21.188762 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:21.188737    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baaaba03-d432-4125-9978-7d63d9f7e5ca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls\" (UID: \"baaaba03-d432-4125-9978-7d63d9f7e5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"
Sep 11 15:01:21.390463 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:21.390424    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"
Sep 11 15:01:21.669467 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:21.669441    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"]
Sep 11 15:01:21.670957 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:01:21.670929    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaaaba03_d432_4125_9978_7d63d9f7e5ca.slice/crio-2d94f83c53cb5b126d348b5782b4ceaf143cff1ca0ead65cd1396e547d07d08e WatchSource:0}: Error finding container 2d94f83c53cb5b126d348b5782b4ceaf143cff1ca0ead65cd1396e547d07d08e: Status 404 returned error can't find the container with id 2d94f83c53cb5b126d348b5782b4ceaf143cff1ca0ead65cd1396e547d07d08e
Sep 11 15:01:22.117488 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:22.117449    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls" event={"ID":"baaaba03-d432-4125-9978-7d63d9f7e5ca","Type":"ContainerStarted","Data":"2d94f83c53cb5b126d348b5782b4ceaf143cff1ca0ead65cd1396e547d07d08e"}
Sep 11 15:01:23.124299 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:23.124270    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls" event={"ID":"baaaba03-d432-4125-9978-7d63d9f7e5ca","Type":"ContainerStarted","Data":"c8fa8ebd9e26d4f9e6e5f4a4a0edeba7df74e763785f4200f2d3aead9614d4bd"}
Sep 11 15:01:23.124691 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:23.124370    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"
Sep 11 15:01:23.144279 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:23.144210    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls" podStartSLOduration=2.306307834 podStartE2EDuration="3.144194897s" totalImagesPullingTime="837.887063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:01:23.143485897 +0000 UTC m=+1262.931750195" watchObservedRunningTime="2026-09-11 15:01:23.144194897 +0000 UTC m=+1262.932459197"
Sep 11 15:01:29.135549 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:29.135519    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"
Sep 11 15:01:43.062801 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:43.062764    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"]
Sep 11 15:01:58.082414 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.082372    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q" podUID="eb779e25-14e3-4e2a-9526-52fe2882bd13" containerName="main" containerID="cri-o://d4fc1b85321aef34738d76d71fe6ce04de25f050c710dfa9750c9865424ceb86" gracePeriod=45
Sep 11 15:01:58.328140 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.328105    2568 generic.go:356] "Generic (PLEG): container finished" podID="eb779e25-14e3-4e2a-9526-52fe2882bd13" containerID="d4fc1b85321aef34738d76d71fe6ce04de25f050c710dfa9750c9865424ceb86" exitCode=0
Sep 11 15:01:58.328277 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.328176    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q" event={"ID":"eb779e25-14e3-4e2a-9526-52fe2882bd13","Type":"ContainerDied","Data":"d4fc1b85321aef34738d76d71fe6ce04de25f050c710dfa9750c9865424ceb86"}
Sep 11 15:01:58.346409 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.346355    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 15:01:58.442114 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.442082    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-model-cache\") pod \"eb779e25-14e3-4e2a-9526-52fe2882bd13\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") "
Sep 11 15:01:58.442267 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.442142    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb779e25-14e3-4e2a-9526-52fe2882bd13-tls-certs\") pod \"eb779e25-14e3-4e2a-9526-52fe2882bd13\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") "
Sep 11 15:01:58.442267 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.442181    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-dshm\") pod \"eb779e25-14e3-4e2a-9526-52fe2882bd13\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") "
Sep 11 15:01:58.442267 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.442212    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-home\") pod \"eb779e25-14e3-4e2a-9526-52fe2882bd13\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") "
Sep 11 15:01:58.442428 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.442263    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-tmp-dir\") pod \"eb779e25-14e3-4e2a-9526-52fe2882bd13\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") "
Sep 11 15:01:58.442428 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.442312    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxdg5\" (UniqueName: \"kubernetes.io/projected/eb779e25-14e3-4e2a-9526-52fe2882bd13-kube-api-access-pxdg5\") pod \"eb779e25-14e3-4e2a-9526-52fe2882bd13\" (UID: \"eb779e25-14e3-4e2a-9526-52fe2882bd13\") "
Sep 11 15:01:58.442428 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.442353    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb779e25-14e3-4e2a-9526-52fe2882bd13" (UID: "eb779e25-14e3-4e2a-9526-52fe2882bd13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:01:58.442590 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.442534    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-home" (OuterVolumeSpecName: "home") pod "eb779e25-14e3-4e2a-9526-52fe2882bd13" (UID: "eb779e25-14e3-4e2a-9526-52fe2882bd13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:01:58.442590 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.442551    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb779e25-14e3-4e2a-9526-52fe2882bd13" (UID: "eb779e25-14e3-4e2a-9526-52fe2882bd13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:01:58.442720 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.442703    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-home\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:01:58.442785 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.442724    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-tmp-dir\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:01:58.442785 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.442755    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-model-cache\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:01:58.444378 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.444352    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-dshm" (OuterVolumeSpecName: "dshm") pod "eb779e25-14e3-4e2a-9526-52fe2882bd13" (UID: "eb779e25-14e3-4e2a-9526-52fe2882bd13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:01:58.444500 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.444480    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb779e25-14e3-4e2a-9526-52fe2882bd13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb779e25-14e3-4e2a-9526-52fe2882bd13" (UID: "eb779e25-14e3-4e2a-9526-52fe2882bd13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:01:58.444500 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.444475    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb779e25-14e3-4e2a-9526-52fe2882bd13-kube-api-access-pxdg5" (OuterVolumeSpecName: "kube-api-access-pxdg5") pod "eb779e25-14e3-4e2a-9526-52fe2882bd13" (UID: "eb779e25-14e3-4e2a-9526-52fe2882bd13"). InnerVolumeSpecName "kube-api-access-pxdg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:01:58.544112 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.544081    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb779e25-14e3-4e2a-9526-52fe2882bd13-tls-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:01:58.544112 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.544106    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb779e25-14e3-4e2a-9526-52fe2882bd13-dshm\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:01:58.544112 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:58.544116    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxdg5\" (UniqueName: \"kubernetes.io/projected/eb779e25-14e3-4e2a-9526-52fe2882bd13-kube-api-access-pxdg5\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:01:59.334777 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:59.334741    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q" event={"ID":"eb779e25-14e3-4e2a-9526-52fe2882bd13","Type":"ContainerDied","Data":"9b91b38f4a150ab98e8952a68631670153420606fa6ddc31cd05ac5c815139c0"}
Sep 11 15:01:59.334777 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:59.334769    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"
Sep 11 15:01:59.335294 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:59.334791    2568 scope.go:117] "RemoveContainer" containerID="d4fc1b85321aef34738d76d71fe6ce04de25f050c710dfa9750c9865424ceb86"
Sep 11 15:01:59.353581 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:59.353556    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"]
Sep 11 15:01:59.358416 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:01:59.358394    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-837632/e2e-conflict-b-372348b4-kserve-76cd4457db-fkl9q"]
Sep 11 15:02:00.662325 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:00.662289    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb779e25-14e3-4e2a-9526-52fe2882bd13" path="/var/lib/kubelet/pods/eb779e25-14e3-4e2a-9526-52fe2882bd13/volumes"
Sep 11 15:02:20.575255 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.575216    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"]
Sep 11 15:02:20.576493 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.576466    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb779e25-14e3-4e2a-9526-52fe2882bd13" containerName="main"
Sep 11 15:02:20.576650 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.576636    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb779e25-14e3-4e2a-9526-52fe2882bd13" containerName="main"
Sep 11 15:02:20.576935 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.576918    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb779e25-14e3-4e2a-9526-52fe2882bd13" containerName="main"
Sep 11 15:02:20.580758 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.580737    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:20.588428 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.588013    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-48898f\"/\"kube-root-ca.crt\""
Sep 11 15:02:20.588428 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.588249    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-48898f\"/\"default-dockercfg-xslvr\""
Sep 11 15:02:20.588784 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.588754    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-48898f\"/\"openshift-service-ca.crt\""
Sep 11 15:02:20.589275 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.589254    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-48898f\"/\"auth-test-model-48898f-kserve-self-signed-certs\""
Sep 11 15:02:20.591044 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.591023    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"]
Sep 11 15:02:20.646403 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.646371    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-tmp-dir\") pod \"auth-test-model-48898f-kserve-86b799c59f-8tgv5\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") " pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:20.646403 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.646405    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-dshm\") pod \"auth-test-model-48898f-kserve-86b799c59f-8tgv5\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") " pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:20.646595 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.646434    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdvn6\" (UniqueName: \"kubernetes.io/projected/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-kube-api-access-kdvn6\") pod \"auth-test-model-48898f-kserve-86b799c59f-8tgv5\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") " pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:20.646595 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.646516    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-home\") pod \"auth-test-model-48898f-kserve-86b799c59f-8tgv5\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") " pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:20.646595 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.646535    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-model-cache\") pod \"auth-test-model-48898f-kserve-86b799c59f-8tgv5\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") " pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:20.646595 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.646585    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-tls-certs\") pod \"auth-test-model-48898f-kserve-86b799c59f-8tgv5\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") " pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:20.747515 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.747478    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdvn6\" (UniqueName: \"kubernetes.io/projected/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-kube-api-access-kdvn6\") pod \"auth-test-model-48898f-kserve-86b799c59f-8tgv5\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") " pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:20.747681 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.747642    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-home\") pod \"auth-test-model-48898f-kserve-86b799c59f-8tgv5\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") " pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:20.747724 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.747678    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-model-cache\") pod \"auth-test-model-48898f-kserve-86b799c59f-8tgv5\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") " pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:20.747765 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.747747    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-tls-certs\") pod \"auth-test-model-48898f-kserve-86b799c59f-8tgv5\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") " pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:20.747808 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.747778    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-tmp-dir\") pod \"auth-test-model-48898f-kserve-86b799c59f-8tgv5\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") " pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:20.747808 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.747802    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-dshm\") pod \"auth-test-model-48898f-kserve-86b799c59f-8tgv5\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") " pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:20.748152 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.748126    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-model-cache\") pod \"auth-test-model-48898f-kserve-86b799c59f-8tgv5\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") " pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:20.748240 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.748152    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-home\") pod \"auth-test-model-48898f-kserve-86b799c59f-8tgv5\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") " pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:20.748240 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.748175    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-tmp-dir\") pod \"auth-test-model-48898f-kserve-86b799c59f-8tgv5\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") " pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:20.750032 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.750006    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-dshm\") pod \"auth-test-model-48898f-kserve-86b799c59f-8tgv5\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") " pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:20.750310 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.750293    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-tls-certs\") pod \"auth-test-model-48898f-kserve-86b799c59f-8tgv5\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") " pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:20.755548 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.755521    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdvn6\" (UniqueName: \"kubernetes.io/projected/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-kube-api-access-kdvn6\") pod \"auth-test-model-48898f-kserve-86b799c59f-8tgv5\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") " pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:20.897811 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:20.897718    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:21.181414 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:21.181326    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"]
Sep 11 15:02:21.183997 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:02:21.183965    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0903a7b6_7261_49d0_b3fc_f839f9f8ecdf.slice/crio-b4703416bc414129d0410f4712671e2b1286e03a7af1e1b4c74b8c2b1d45e2a6 WatchSource:0}: Error finding container b4703416bc414129d0410f4712671e2b1286e03a7af1e1b4c74b8c2b1d45e2a6: Status 404 returned error can't find the container with id b4703416bc414129d0410f4712671e2b1286e03a7af1e1b4c74b8c2b1d45e2a6
Sep 11 15:02:21.453338 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:21.453247    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5" event={"ID":"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf","Type":"ContainerStarted","Data":"39b7eaa1504230d58c0ecb519c66a56ececde15e37609e4ad013a86d459b66ff"}
Sep 11 15:02:21.453338 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:21.453296    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5" event={"ID":"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf","Type":"ContainerStarted","Data":"b4703416bc414129d0410f4712671e2b1286e03a7af1e1b4c74b8c2b1d45e2a6"}
Sep 11 15:02:21.473973 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:21.473916    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5" podStartSLOduration=1.473896616 podStartE2EDuration="1.473896616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:02:21.471154837 +0000 UTC m=+1321.259419137" watchObservedRunningTime="2026-09-11 15:02:21.473896616 +0000 UTC m=+1321.262160932"
Sep 11 15:02:30.898058 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:30.898018    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:30.898518 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:30.898075    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:30.910568 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:30.910545    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:31.518995 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:31.518952    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:02:41.076959 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.076918    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h"]
Sep 11 15:02:41.077551 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.077172    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h" podUID="ce2d526b-0807-4b87-a14c-1f3ac57d14a4" containerName="payload-processing" containerID="cri-o://d02300f3ba689c39be68c891e2a8171a0f364e53eec4f1436d49f55f7eb2d328" gracePeriod=30
Sep 11 15:02:41.081648 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.081621    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp"]
Sep 11 15:02:41.082061 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.082013    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp" podUID="f46167db-ae60-4728-a579-e69655ced01e" containerName="payload-pre-processing" containerID="cri-o://e840b5b0474dadbe502951626a6147d926551428157241df8e761a400d3fdc6b" gracePeriod=30
Sep 11 15:02:41.367865 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.367839    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h"
Sep 11 15:02:41.380207 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.380183    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp"
Sep 11 15:02:41.458042 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.458008    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce2d526b-0807-4b87-a14c-1f3ac57d14a4-plugins-config-volume\") pod \"ce2d526b-0807-4b87-a14c-1f3ac57d14a4\" (UID: \"ce2d526b-0807-4b87-a14c-1f3ac57d14a4\") "
Sep 11 15:02:41.458240 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.458055    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pftm\" (UniqueName: \"kubernetes.io/projected/ce2d526b-0807-4b87-a14c-1f3ac57d14a4-kube-api-access-4pftm\") pod \"ce2d526b-0807-4b87-a14c-1f3ac57d14a4\" (UID: \"ce2d526b-0807-4b87-a14c-1f3ac57d14a4\") "
Sep 11 15:02:41.458240 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.458081    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f46167db-ae60-4728-a579-e69655ced01e-plugins-config-volume\") pod \"f46167db-ae60-4728-a579-e69655ced01e\" (UID: \"f46167db-ae60-4728-a579-e69655ced01e\") "
Sep 11 15:02:41.458240 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.458110    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbj22\" (UniqueName: \"kubernetes.io/projected/f46167db-ae60-4728-a579-e69655ced01e-kube-api-access-fbj22\") pod \"f46167db-ae60-4728-a579-e69655ced01e\" (UID: \"f46167db-ae60-4728-a579-e69655ced01e\") "
Sep 11 15:02:41.458463 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.458427    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46167db-ae60-4728-a579-e69655ced01e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f46167db-ae60-4728-a579-e69655ced01e" (UID: "f46167db-ae60-4728-a579-e69655ced01e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:02:41.458578 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.458459    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2d526b-0807-4b87-a14c-1f3ac57d14a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ce2d526b-0807-4b87-a14c-1f3ac57d14a4" (UID: "ce2d526b-0807-4b87-a14c-1f3ac57d14a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:02:41.460282 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.460256    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46167db-ae60-4728-a579-e69655ced01e-kube-api-access-fbj22" (OuterVolumeSpecName: "kube-api-access-fbj22") pod "f46167db-ae60-4728-a579-e69655ced01e" (UID: "f46167db-ae60-4728-a579-e69655ced01e"). InnerVolumeSpecName "kube-api-access-fbj22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:02:41.460282 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.460272    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2d526b-0807-4b87-a14c-1f3ac57d14a4-kube-api-access-4pftm" (OuterVolumeSpecName: "kube-api-access-4pftm") pod "ce2d526b-0807-4b87-a14c-1f3ac57d14a4" (UID: "ce2d526b-0807-4b87-a14c-1f3ac57d14a4"). InnerVolumeSpecName "kube-api-access-4pftm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:02:41.558952 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.558921    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce2d526b-0807-4b87-a14c-1f3ac57d14a4-plugins-config-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:02:41.558952 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.558949    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pftm\" (UniqueName: \"kubernetes.io/projected/ce2d526b-0807-4b87-a14c-1f3ac57d14a4-kube-api-access-4pftm\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:02:41.558952 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.558959    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f46167db-ae60-4728-a579-e69655ced01e-plugins-config-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:02:41.559180 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.558968    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbj22\" (UniqueName: \"kubernetes.io/projected/f46167db-ae60-4728-a579-e69655ced01e-kube-api-access-fbj22\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:02:41.573993 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.573957    2568 generic.go:356] "Generic (PLEG): container finished" podID="f46167db-ae60-4728-a579-e69655ced01e" containerID="e840b5b0474dadbe502951626a6147d926551428157241df8e761a400d3fdc6b" exitCode=0
Sep 11 15:02:41.574128 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.574009    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp"
Sep 11 15:02:41.574128 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.574038    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp" event={"ID":"f46167db-ae60-4728-a579-e69655ced01e","Type":"ContainerDied","Data":"e840b5b0474dadbe502951626a6147d926551428157241df8e761a400d3fdc6b"}
Sep 11 15:02:41.574128 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.574091    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp" event={"ID":"f46167db-ae60-4728-a579-e69655ced01e","Type":"ContainerDied","Data":"785300fae4fdf2a319479db13733ef086cda9844ca9d1905257421718af27da9"}
Sep 11 15:02:41.574128 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.574115    2568 scope.go:117] "RemoveContainer" containerID="e840b5b0474dadbe502951626a6147d926551428157241df8e761a400d3fdc6b"
Sep 11 15:02:41.575562 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.575538    2568 generic.go:356] "Generic (PLEG): container finished" podID="ce2d526b-0807-4b87-a14c-1f3ac57d14a4" containerID="d02300f3ba689c39be68c891e2a8171a0f364e53eec4f1436d49f55f7eb2d328" exitCode=0
Sep 11 15:02:41.575688 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.575586    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h"
Sep 11 15:02:41.575688 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.575587    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h" event={"ID":"ce2d526b-0807-4b87-a14c-1f3ac57d14a4","Type":"ContainerDied","Data":"d02300f3ba689c39be68c891e2a8171a0f364e53eec4f1436d49f55f7eb2d328"}
Sep 11 15:02:41.575688 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.575644    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h" event={"ID":"ce2d526b-0807-4b87-a14c-1f3ac57d14a4","Type":"ContainerDied","Data":"4d5623ad2bc0d7e3707ea15a31a5333a8bcc6fe4d153c1bfd3a8ad2455251a64"}
Sep 11 15:02:41.582880 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.582862    2568 scope.go:117] "RemoveContainer" containerID="e840b5b0474dadbe502951626a6147d926551428157241df8e761a400d3fdc6b"
Sep 11 15:02:41.583145 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:02:41.583123    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e840b5b0474dadbe502951626a6147d926551428157241df8e761a400d3fdc6b\": container with ID starting with e840b5b0474dadbe502951626a6147d926551428157241df8e761a400d3fdc6b not found: ID does not exist" containerID="e840b5b0474dadbe502951626a6147d926551428157241df8e761a400d3fdc6b"
Sep 11 15:02:41.583234 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.583156    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e840b5b0474dadbe502951626a6147d926551428157241df8e761a400d3fdc6b"} err="failed to get container status \"e840b5b0474dadbe502951626a6147d926551428157241df8e761a400d3fdc6b\": rpc error: code = NotFound desc = could not find container \"e840b5b0474dadbe502951626a6147d926551428157241df8e761a400d3fdc6b\": container with ID starting with e840b5b0474dadbe502951626a6147d926551428157241df8e761a400d3fdc6b not found: ID does not exist"
Sep 11 15:02:41.583234 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.583180    2568 scope.go:117] "RemoveContainer" containerID="d02300f3ba689c39be68c891e2a8171a0f364e53eec4f1436d49f55f7eb2d328"
Sep 11 15:02:41.591190 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.591171    2568 scope.go:117] "RemoveContainer" containerID="d02300f3ba689c39be68c891e2a8171a0f364e53eec4f1436d49f55f7eb2d328"
Sep 11 15:02:41.591425 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:02:41.591408    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d02300f3ba689c39be68c891e2a8171a0f364e53eec4f1436d49f55f7eb2d328\": container with ID starting with d02300f3ba689c39be68c891e2a8171a0f364e53eec4f1436d49f55f7eb2d328 not found: ID does not exist" containerID="d02300f3ba689c39be68c891e2a8171a0f364e53eec4f1436d49f55f7eb2d328"
Sep 11 15:02:41.591473 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.591432    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02300f3ba689c39be68c891e2a8171a0f364e53eec4f1436d49f55f7eb2d328"} err="failed to get container status \"d02300f3ba689c39be68c891e2a8171a0f364e53eec4f1436d49f55f7eb2d328\": rpc error: code = NotFound desc = could not find container \"d02300f3ba689c39be68c891e2a8171a0f364e53eec4f1436d49f55f7eb2d328\": container with ID starting with d02300f3ba689c39be68c891e2a8171a0f364e53eec4f1436d49f55f7eb2d328 not found: ID does not exist"
Sep 11 15:02:41.600669 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.600621    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h"]
Sep 11 15:02:41.604921 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.604900    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a843ae03-5887db69b9-jw82h"]
Sep 11 15:02:41.617152 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.617103    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp"]
Sep 11 15:02:41.620517 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:41.620494    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a843ae03-6d7c45b9c4-6vwhp"]
Sep 11 15:02:42.668470 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:42.668414    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce2d526b-0807-4b87-a14c-1f3ac57d14a4" path="/var/lib/kubelet/pods/ce2d526b-0807-4b87-a14c-1f3ac57d14a4/volumes"
Sep 11 15:02:42.669436 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:02:42.669398    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f46167db-ae60-4728-a579-e69655ced01e" path="/var/lib/kubelet/pods/f46167db-ae60-4728-a579-e69655ced01e/volumes"
Sep 11 15:03:59.643262 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:03:59.643229    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"]
Sep 11 15:03:59.643747 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:03:59.643471    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9" podUID="c5afaa36-a4f1-4239-9e4d-a071ba934f99" containerName="istio-proxy" containerID="cri-o://23e9c1bde7c407527d40453b8d380c5e791ac6a8b012cd74ee36dbf65d4a4b5c" gracePeriod=30
Sep 11 15:04:04.909880 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.909799    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 15:04:04.977186 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.977157    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-token\") pod \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") "
Sep 11 15:04:04.977344 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.977197    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-podinfo\") pod \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") "
Sep 11 15:04:04.977344 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.977269    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-data\") pod \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") "
Sep 11 15:04:04.977344 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.977301    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istiod-ca-cert\") pod \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") "
Sep 11 15:04:04.977344 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.977339    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lw98q\" (UniqueName: \"kubernetes.io/projected/c5afaa36-a4f1-4239-9e4d-a071ba934f99-kube-api-access-lw98q\") pod \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") "
Sep 11 15:04:04.977548 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.977379    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-credential-socket\") pod \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") "
Sep 11 15:04:04.977548 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.977406    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-workload-socket\") pod \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") "
Sep 11 15:04:04.977548 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.977460    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-envoy\") pod \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") "
Sep 11 15:04:04.977548 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.977489    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-workload-certs\") pod \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\" (UID: \"c5afaa36-a4f1-4239-9e4d-a071ba934f99\") "
Sep 11 15:04:04.977753 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.977588    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-data" (OuterVolumeSpecName: "istio-data") pod "c5afaa36-a4f1-4239-9e4d-a071ba934f99" (UID: "c5afaa36-a4f1-4239-9e4d-a071ba934f99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:04:04.977753 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.977649    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c5afaa36-a4f1-4239-9e4d-a071ba934f99" (UID: "c5afaa36-a4f1-4239-9e4d-a071ba934f99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:04:04.977897 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.977850    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c5afaa36-a4f1-4239-9e4d-a071ba934f99" (UID: "c5afaa36-a4f1-4239-9e4d-a071ba934f99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:04:04.977897 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.977873    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c5afaa36-a4f1-4239-9e4d-a071ba934f99" (UID: "c5afaa36-a4f1-4239-9e4d-a071ba934f99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:04:04.978009 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.977876    2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-data\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:04:04.978009 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.977924    2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-credential-socket\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:04:04.978009 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.977892    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c5afaa36-a4f1-4239-9e4d-a071ba934f99" (UID: "c5afaa36-a4f1-4239-9e4d-a071ba934f99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:04:04.979640 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.979614    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c5afaa36-a4f1-4239-9e4d-a071ba934f99" (UID: "c5afaa36-a4f1-4239-9e4d-a071ba934f99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:04:04.979776 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.979668    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c5afaa36-a4f1-4239-9e4d-a071ba934f99" (UID: "c5afaa36-a4f1-4239-9e4d-a071ba934f99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 15:04:04.979776 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.979716    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-token" (OuterVolumeSpecName: "istio-token") pod "c5afaa36-a4f1-4239-9e4d-a071ba934f99" (UID: "c5afaa36-a4f1-4239-9e4d-a071ba934f99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:04:04.979776 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:04.979731    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5afaa36-a4f1-4239-9e4d-a071ba934f99-kube-api-access-lw98q" (OuterVolumeSpecName: "kube-api-access-lw98q") pod "c5afaa36-a4f1-4239-9e4d-a071ba934f99" (UID: "c5afaa36-a4f1-4239-9e4d-a071ba934f99"). InnerVolumeSpecName "kube-api-access-lw98q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:04:05.017637 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:05.017604    2568 generic.go:356] "Generic (PLEG): container finished" podID="c5afaa36-a4f1-4239-9e4d-a071ba934f99" containerID="23e9c1bde7c407527d40453b8d380c5e791ac6a8b012cd74ee36dbf65d4a4b5c" exitCode=0
Sep 11 15:04:05.017770 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:05.017649    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"
Sep 11 15:04:05.017770 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:05.017659    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9" event={"ID":"c5afaa36-a4f1-4239-9e4d-a071ba934f99","Type":"ContainerDied","Data":"23e9c1bde7c407527d40453b8d380c5e791ac6a8b012cd74ee36dbf65d4a4b5c"}
Sep 11 15:04:05.017770 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:05.017707    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9" event={"ID":"c5afaa36-a4f1-4239-9e4d-a071ba934f99","Type":"ContainerDied","Data":"a13cc65e3f23a52d9c791b93ac418477b5748c68c02e2792161e77ffbe5a2cdd"}
Sep 11 15:04:05.017770 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:05.017733    2568 scope.go:117] "RemoveContainer" containerID="23e9c1bde7c407527d40453b8d380c5e791ac6a8b012cd74ee36dbf65d4a4b5c"
Sep 11 15:04:05.028267 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:05.028251    2568 scope.go:117] "RemoveContainer" containerID="e51541494a1ccf6b0758638122e87b687ffbd1b3f4a0827f2eabd1c7608744b7"
Sep 11 15:04:05.038810 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:05.038784    2568 scope.go:117] "RemoveContainer" containerID="23e9c1bde7c407527d40453b8d380c5e791ac6a8b012cd74ee36dbf65d4a4b5c"
Sep 11 15:04:05.039164 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:04:05.039138    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23e9c1bde7c407527d40453b8d380c5e791ac6a8b012cd74ee36dbf65d4a4b5c\": container with ID starting with 23e9c1bde7c407527d40453b8d380c5e791ac6a8b012cd74ee36dbf65d4a4b5c not found: ID does not exist" containerID="23e9c1bde7c407527d40453b8d380c5e791ac6a8b012cd74ee36dbf65d4a4b5c"
Sep 11 15:04:05.039234 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:05.039175    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23e9c1bde7c407527d40453b8d380c5e791ac6a8b012cd74ee36dbf65d4a4b5c"} err="failed to get container status \"23e9c1bde7c407527d40453b8d380c5e791ac6a8b012cd74ee36dbf65d4a4b5c\": rpc error: code = NotFound desc = could not find container \"23e9c1bde7c407527d40453b8d380c5e791ac6a8b012cd74ee36dbf65d4a4b5c\": container with ID starting with 23e9c1bde7c407527d40453b8d380c5e791ac6a8b012cd74ee36dbf65d4a4b5c not found: ID does not exist"
Sep 11 15:04:05.039234 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:05.039199    2568 scope.go:117] "RemoveContainer" containerID="e51541494a1ccf6b0758638122e87b687ffbd1b3f4a0827f2eabd1c7608744b7"
Sep 11 15:04:05.039522 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:04:05.039502    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e51541494a1ccf6b0758638122e87b687ffbd1b3f4a0827f2eabd1c7608744b7\": container with ID starting with e51541494a1ccf6b0758638122e87b687ffbd1b3f4a0827f2eabd1c7608744b7 not found: ID does not exist" containerID="e51541494a1ccf6b0758638122e87b687ffbd1b3f4a0827f2eabd1c7608744b7"
Sep 11 15:04:05.039564 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:05.039529    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e51541494a1ccf6b0758638122e87b687ffbd1b3f4a0827f2eabd1c7608744b7"} err="failed to get container status \"e51541494a1ccf6b0758638122e87b687ffbd1b3f4a0827f2eabd1c7608744b7\": rpc error: code = NotFound desc = could not find container \"e51541494a1ccf6b0758638122e87b687ffbd1b3f4a0827f2eabd1c7608744b7\": container with ID starting with e51541494a1ccf6b0758638122e87b687ffbd1b3f4a0827f2eabd1c7608744b7 not found: ID does not exist"
Sep 11 15:04:05.041040 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:05.041019    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"]
Sep 11 15:04:05.044871 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:05.044849    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a843ae03-openshift-default-7c48d8b9cb-lpqb9"]
Sep 11 15:04:05.078487 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:05.078463    2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-podinfo\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:04:05.078487 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:05.078485    2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istiod-ca-cert\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:04:05.078623 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:05.078496    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw98q\" (UniqueName: \"kubernetes.io/projected/c5afaa36-a4f1-4239-9e4d-a071ba934f99-kube-api-access-lw98q\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:04:05.078623 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:05.078505    2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-workload-socket\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:04:05.078623 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:05.078515    2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-envoy\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:04:05.078623 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:05.078523    2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5afaa36-a4f1-4239-9e4d-a071ba934f99-workload-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:04:05.078623 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:05.078530    2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5afaa36-a4f1-4239-9e4d-a071ba934f99-istio-token\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:04:06.663011 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:06.662978    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5afaa36-a4f1-4239-9e4d-a071ba934f99" path="/var/lib/kubelet/pods/c5afaa36-a4f1-4239-9e4d-a071ba934f99/volumes"
Sep 11 15:04:09.477920 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.477886    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"]
Sep 11 15:04:09.478706 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.478623    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5afaa36-a4f1-4239-9e4d-a071ba934f99" containerName="istio-proxy"
Sep 11 15:04:09.478706 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.478648    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5afaa36-a4f1-4239-9e4d-a071ba934f99" containerName="istio-proxy"
Sep 11 15:04:09.478706 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.478671    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f46167db-ae60-4728-a579-e69655ced01e" containerName="payload-pre-processing"
Sep 11 15:04:09.478706 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.478681    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f46167db-ae60-4728-a579-e69655ced01e" containerName="payload-pre-processing"
Sep 11 15:04:09.478963 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.478721    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce2d526b-0807-4b87-a14c-1f3ac57d14a4" containerName="payload-processing"
Sep 11 15:04:09.478963 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.478732    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce2d526b-0807-4b87-a14c-1f3ac57d14a4" containerName="payload-processing"
Sep 11 15:04:09.478963 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.478850    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce2d526b-0807-4b87-a14c-1f3ac57d14a4" containerName="payload-processing"
Sep 11 15:04:09.478963 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.478867    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f46167db-ae60-4728-a579-e69655ced01e" containerName="payload-pre-processing"
Sep 11 15:04:09.478963 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.478876    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5afaa36-a4f1-4239-9e4d-a071ba934f99" containerName="istio-proxy"
Sep 11 15:04:09.483631 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.483615    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:09.486661 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.486641    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e7ca56\"/\"openshift-service-ca.crt\""
Sep 11 15:04:09.487900 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.487874    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e7ca56\"/\"embedding-w0-e7ca56-kserve-self-signed-certs\""
Sep 11 15:04:09.488012 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.487885    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e7ca56\"/\"default-dockercfg-rhxhm\""
Sep 11 15:04:09.488421 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.488401    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e7ca56\"/\"kube-root-ca.crt\""
Sep 11 15:04:09.490935 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.490914    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"]
Sep 11 15:04:09.521597 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.521572    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-model-cache\") pod \"embedding-w0-e7ca56-kserve-7c4987db85-2gbq2\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") " pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:09.521735 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.521601    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-dshm\") pod \"embedding-w0-e7ca56-kserve-7c4987db85-2gbq2\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") " pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:09.521735 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.521631    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq8jl\" (UniqueName: \"kubernetes.io/projected/cb807d22-6c25-4b59-99b5-b79df420519c-kube-api-access-sq8jl\") pod \"embedding-w0-e7ca56-kserve-7c4987db85-2gbq2\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") " pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:09.521735 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.521655    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb807d22-6c25-4b59-99b5-b79df420519c-tls-certs\") pod \"embedding-w0-e7ca56-kserve-7c4987db85-2gbq2\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") " pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:09.521919 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.521726    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-home\") pod \"embedding-w0-e7ca56-kserve-7c4987db85-2gbq2\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") " pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:09.521919 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.521768    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-tmp-dir\") pod \"embedding-w0-e7ca56-kserve-7c4987db85-2gbq2\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") " pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:09.622452 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.622419    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb807d22-6c25-4b59-99b5-b79df420519c-tls-certs\") pod \"embedding-w0-e7ca56-kserve-7c4987db85-2gbq2\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") " pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:09.622645 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.622465    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-home\") pod \"embedding-w0-e7ca56-kserve-7c4987db85-2gbq2\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") " pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:09.622645 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.622490    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-tmp-dir\") pod \"embedding-w0-e7ca56-kserve-7c4987db85-2gbq2\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") " pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:09.622645 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.622557    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-model-cache\") pod \"embedding-w0-e7ca56-kserve-7c4987db85-2gbq2\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") " pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:09.622645 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.622581    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-dshm\") pod \"embedding-w0-e7ca56-kserve-7c4987db85-2gbq2\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") " pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:09.622645 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.622622    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq8jl\" (UniqueName: \"kubernetes.io/projected/cb807d22-6c25-4b59-99b5-b79df420519c-kube-api-access-sq8jl\") pod \"embedding-w0-e7ca56-kserve-7c4987db85-2gbq2\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") " pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:09.622943 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.622914    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-home\") pod \"embedding-w0-e7ca56-kserve-7c4987db85-2gbq2\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") " pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:09.622943 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.622934    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-model-cache\") pod \"embedding-w0-e7ca56-kserve-7c4987db85-2gbq2\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") " pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:09.623059 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.622971    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-tmp-dir\") pod \"embedding-w0-e7ca56-kserve-7c4987db85-2gbq2\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") " pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:09.624776 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.624753    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-dshm\") pod \"embedding-w0-e7ca56-kserve-7c4987db85-2gbq2\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") " pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:09.625080 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.625062    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb807d22-6c25-4b59-99b5-b79df420519c-tls-certs\") pod \"embedding-w0-e7ca56-kserve-7c4987db85-2gbq2\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") " pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:09.630394 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.630375    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq8jl\" (UniqueName: \"kubernetes.io/projected/cb807d22-6c25-4b59-99b5-b79df420519c-kube-api-access-sq8jl\") pod \"embedding-w0-e7ca56-kserve-7c4987db85-2gbq2\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") " pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:09.794748 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:09.794712    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:10.070841 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:10.070795    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"]
Sep 11 15:04:10.072380 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:04:10.072353    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb807d22_6c25_4b59_99b5_b79df420519c.slice/crio-282f4b67732f4ae93442c2f668efab8cf26108b73c9b4b8a992e003a1cad1349 WatchSource:0}: Error finding container 282f4b67732f4ae93442c2f668efab8cf26108b73c9b4b8a992e003a1cad1349: Status 404 returned error can't find the container with id 282f4b67732f4ae93442c2f668efab8cf26108b73c9b4b8a992e003a1cad1349
Sep 11 15:04:11.054039 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:11.054004    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2" event={"ID":"cb807d22-6c25-4b59-99b5-b79df420519c","Type":"ContainerStarted","Data":"72bddf920e40c72e067c4897cf791f4813cfa16a4975b55c2a0bc3065dcef01e"}
Sep 11 15:04:11.054474 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:11.054051    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2" event={"ID":"cb807d22-6c25-4b59-99b5-b79df420519c","Type":"ContainerStarted","Data":"282f4b67732f4ae93442c2f668efab8cf26108b73c9b4b8a992e003a1cad1349"}
Sep 11 15:04:11.073747 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:11.073697    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2" podStartSLOduration=2.0736812159999998 podStartE2EDuration="2.073681216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:04:11.071855876 +0000 UTC m=+1430.860120183" watchObservedRunningTime="2026-09-11 15:04:11.073681216 +0000 UTC m=+1430.861945514"
Sep 11 15:04:19.795266 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:19.795170    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:19.795266 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:19.795238    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:19.808128 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:19.808100    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:20.113109 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:20.113011    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:04:55.624508 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.624469    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"]
Sep 11 15:04:55.625565 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.625219    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5afaa36-a4f1-4239-9e4d-a071ba934f99" containerName="istio-proxy"
Sep 11 15:04:55.625565 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.625537    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5afaa36-a4f1-4239-9e4d-a071ba934f99" containerName="istio-proxy"
Sep 11 15:04:55.625769 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.625748    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5afaa36-a4f1-4239-9e4d-a071ba934f99" containerName="istio-proxy"
Sep 11 15:04:55.629321 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.629296    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.631968 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.631945    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-54b7f9-openshift-default-dockercfg-bj6jm\""
Sep 11 15:04:55.643879 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.643855    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"]
Sep 11 15:04:55.776491 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.776449    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-podinfo\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.776491 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.776498    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e993b0e-b03c-4eca-b18b-91356ca73783-istiod-ca-cert\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.776720 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.776525    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-workload-certs\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.776720 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.776585    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-token\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.776720 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.776643    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-credential-socket\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.776720 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.776674    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-envoy\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.776720 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.776703    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86s7d\" (UniqueName: \"kubernetes.io/projected/6e993b0e-b03c-4eca-b18b-91356ca73783-kube-api-access-86s7d\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.776935 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.776796    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-data\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.776935 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.776841    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-workload-socket\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.877416 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.877324    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-envoy\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.877416 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.877369    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86s7d\" (UniqueName: \"kubernetes.io/projected/6e993b0e-b03c-4eca-b18b-91356ca73783-kube-api-access-86s7d\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.877645 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.877466    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-data\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.877645 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.877497    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-workload-socket\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.877645 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.877548    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-podinfo\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.877645 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.877589    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e993b0e-b03c-4eca-b18b-91356ca73783-istiod-ca-cert\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.877645 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.877629    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-workload-certs\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.877957 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.877663    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-token\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.877957 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.877696    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-credential-socket\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.877957 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.877906    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-data\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.877957 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.877946    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-workload-socket\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.878330 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.878309    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-workload-certs\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.878393 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.878339    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-credential-socket\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.878876 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.878850    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e993b0e-b03c-4eca-b18b-91356ca73783-istiod-ca-cert\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.879685 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.879665    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-envoy\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.880083 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.880062    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-podinfo\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.888231 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.888209    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-token\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.888531 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.888509    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86s7d\" (UniqueName: \"kubernetes.io/projected/6e993b0e-b03c-4eca-b18b-91356ca73783-kube-api-access-86s7d\") pod \"e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") " pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:55.942574 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:55.942545    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:56.222449 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:56.222423    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"]
Sep 11 15:04:56.224431 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:04:56.224404    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e993b0e_b03c_4eca_b18b_91356ca73783.slice/crio-1ee03c51d1c0e63ff80219cf10b8399dc400dd42a648324743654ab248460476 WatchSource:0}: Error finding container 1ee03c51d1c0e63ff80219cf10b8399dc400dd42a648324743654ab248460476: Status 404 returned error can't find the container with id 1ee03c51d1c0e63ff80219cf10b8399dc400dd42a648324743654ab248460476
Sep 11 15:04:56.226852 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:56.226802    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:04:56.226960 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:56.226883    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:04:56.226960 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:56.226910    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:04:56.295749 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:56.295722    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m" event={"ID":"6e993b0e-b03c-4eca-b18b-91356ca73783","Type":"ContainerStarted","Data":"1ee03c51d1c0e63ff80219cf10b8399dc400dd42a648324743654ab248460476"}
Sep 11 15:04:57.303023 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:57.302986    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m" event={"ID":"6e993b0e-b03c-4eca-b18b-91356ca73783","Type":"ContainerStarted","Data":"67a35fe9b1c1c931673ee8821faa0646ab573bd00f8253bda85bf5a3f49b82f2"}
Sep 11 15:04:57.325622 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:57.325571    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m" podStartSLOduration=2.325553639 podStartE2EDuration="2.325553639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:04:57.32426457 +0000 UTC m=+1477.112528894" watchObservedRunningTime="2026-09-11 15:04:57.325553639 +0000 UTC m=+1477.113817938"
Sep 11 15:04:57.943655 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:57.943625    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:57.950018 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:57.949994    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:58.308433 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:58.308405    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:04:58.309375 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:04:58.309348    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:05:05.109547 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:05.109505    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf"]
Sep 11 15:05:05.113888 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:05.113865    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf"
Sep 11 15:05:05.117063 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:05.117044    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-54b7f9-dockercfg-wwqlg\""
Sep 11 15:05:05.118066 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:05.118052    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-54b7f9\""
Sep 11 15:05:05.132788 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:05.132756    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf"]
Sep 11 15:05:05.281860 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:05.281804    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7264546b-16e5-4b54-9542-5943919be700-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf\" (UID: \"7264546b-16e5-4b54-9542-5943919be700\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf"
Sep 11 15:05:05.282033 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:05.281928    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z7cp\" (UniqueName: \"kubernetes.io/projected/7264546b-16e5-4b54-9542-5943919be700-kube-api-access-4z7cp\") pod \"payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf\" (UID: \"7264546b-16e5-4b54-9542-5943919be700\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf"
Sep 11 15:05:05.383356 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:05.383270    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7264546b-16e5-4b54-9542-5943919be700-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf\" (UID: \"7264546b-16e5-4b54-9542-5943919be700\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf"
Sep 11 15:05:05.383507 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:05.383370    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z7cp\" (UniqueName: \"kubernetes.io/projected/7264546b-16e5-4b54-9542-5943919be700-kube-api-access-4z7cp\") pod \"payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf\" (UID: \"7264546b-16e5-4b54-9542-5943919be700\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf"
Sep 11 15:05:05.384167 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:05.384145    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7264546b-16e5-4b54-9542-5943919be700-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf\" (UID: \"7264546b-16e5-4b54-9542-5943919be700\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf"
Sep 11 15:05:05.391883 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:05.391858    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z7cp\" (UniqueName: \"kubernetes.io/projected/7264546b-16e5-4b54-9542-5943919be700-kube-api-access-4z7cp\") pod \"payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf\" (UID: \"7264546b-16e5-4b54-9542-5943919be700\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf"
Sep 11 15:05:05.425519 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:05.425488    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf"
Sep 11 15:05:05.699992 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:05.699962    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf"]
Sep 11 15:05:06.356586 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:06.356552    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf" event={"ID":"7264546b-16e5-4b54-9542-5943919be700","Type":"ContainerStarted","Data":"028765c7cb28f5850004693430f77fd1b9c4991ef9aa9757d9d1fa0b5e76bdc8"}
Sep 11 15:05:06.357066 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:06.356603    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf" event={"ID":"7264546b-16e5-4b54-9542-5943919be700","Type":"ContainerStarted","Data":"ad9e9c5deb6585ec2becf1fff44333f0dcd8b7ba6820d90f8ecc4711572e8ec9"}
Sep 11 15:05:06.357066 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:06.356673    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf"
Sep 11 15:05:06.374801 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:06.374758    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf" podStartSLOduration=1.374743486 podStartE2EDuration="1.374743486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:05:06.372981068 +0000 UTC m=+1486.161245364" watchObservedRunningTime="2026-09-11 15:05:06.374743486 +0000 UTC m=+1486.163007782"
Sep 11 15:05:11.824724 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:11.824691    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"]
Sep 11 15:05:17.365029 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:17.364992    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf"
Sep 11 15:05:23.098142 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.098105    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"]
Sep 11 15:05:23.099248 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.099220    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf" podUID="0cbc975c-be6f-4326-b8a1-b2de058f4086" containerName="maas-api" containerID="cri-o://97a558d8edf0caa73c6220a5bcdcc685ebbe0256f158e160a0112e2fbba274b7" gracePeriod=30
Sep 11 15:05:23.162020 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.161982    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv"]
Sep 11 15:05:23.162258 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.162226    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv" podUID="80136c17-90ca-4571-9164-f25620470160" containerName="payload-processing" containerID="cri-o://4f0945ce85084fa08c576dde6f3952eaec5e417740775d14b8830318864c7305" gracePeriod=30
Sep 11 15:05:23.444175 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.444137    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"
Sep 11 15:05:23.458565 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.458534    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbc975c-be6f-4326-b8a1-b2de058f4086-metrics-certs\") pod \"0cbc975c-be6f-4326-b8a1-b2de058f4086\" (UID: \"0cbc975c-be6f-4326-b8a1-b2de058f4086\") "
Sep 11 15:05:23.458686 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.458577    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbsvf\" (UniqueName: \"kubernetes.io/projected/0cbc975c-be6f-4326-b8a1-b2de058f4086-kube-api-access-mbsvf\") pod \"0cbc975c-be6f-4326-b8a1-b2de058f4086\" (UID: \"0cbc975c-be6f-4326-b8a1-b2de058f4086\") "
Sep 11 15:05:23.458686 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.458634    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbc975c-be6f-4326-b8a1-b2de058f4086-openshift-service-ca\") pod \"0cbc975c-be6f-4326-b8a1-b2de058f4086\" (UID: \"0cbc975c-be6f-4326-b8a1-b2de058f4086\") "
Sep 11 15:05:23.458801 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.458694    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cbc975c-be6f-4326-b8a1-b2de058f4086-maas-api-tls\") pod \"0cbc975c-be6f-4326-b8a1-b2de058f4086\" (UID: \"0cbc975c-be6f-4326-b8a1-b2de058f4086\") "
Sep 11 15:05:23.459092 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.459064    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbc975c-be6f-4326-b8a1-b2de058f4086-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0cbc975c-be6f-4326-b8a1-b2de058f4086" (UID: "0cbc975c-be6f-4326-b8a1-b2de058f4086"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:05:23.460876 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.460850    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cbc975c-be6f-4326-b8a1-b2de058f4086-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0cbc975c-be6f-4326-b8a1-b2de058f4086" (UID: "0cbc975c-be6f-4326-b8a1-b2de058f4086"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:05:23.461000 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.460958    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbc975c-be6f-4326-b8a1-b2de058f4086-kube-api-access-mbsvf" (OuterVolumeSpecName: "kube-api-access-mbsvf") pod "0cbc975c-be6f-4326-b8a1-b2de058f4086" (UID: "0cbc975c-be6f-4326-b8a1-b2de058f4086"). InnerVolumeSpecName "kube-api-access-mbsvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:05:23.461185 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.461151    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cbc975c-be6f-4326-b8a1-b2de058f4086-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0cbc975c-be6f-4326-b8a1-b2de058f4086" (UID: "0cbc975c-be6f-4326-b8a1-b2de058f4086"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:05:23.478509 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.478475    2568 generic.go:356] "Generic (PLEG): container finished" podID="80136c17-90ca-4571-9164-f25620470160" containerID="4f0945ce85084fa08c576dde6f3952eaec5e417740775d14b8830318864c7305" exitCode=0
Sep 11 15:05:23.478654 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.478548    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv" event={"ID":"80136c17-90ca-4571-9164-f25620470160","Type":"ContainerDied","Data":"4f0945ce85084fa08c576dde6f3952eaec5e417740775d14b8830318864c7305"}
Sep 11 15:05:23.478654 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.478598    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv" event={"ID":"80136c17-90ca-4571-9164-f25620470160","Type":"ContainerDied","Data":"7a986c60d64af922b3206cc59f566866d0f31ff111162914e62737da27db115a"}
Sep 11 15:05:23.478654 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.478614    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a986c60d64af922b3206cc59f566866d0f31ff111162914e62737da27db115a"
Sep 11 15:05:23.479910 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.479880    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv"
Sep 11 15:05:23.480156 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.480132    2568 generic.go:356] "Generic (PLEG): container finished" podID="0cbc975c-be6f-4326-b8a1-b2de058f4086" containerID="97a558d8edf0caa73c6220a5bcdcc685ebbe0256f158e160a0112e2fbba274b7" exitCode=0
Sep 11 15:05:23.480259 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.480164    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"
Sep 11 15:05:23.480259 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.480214    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf" event={"ID":"0cbc975c-be6f-4326-b8a1-b2de058f4086","Type":"ContainerDied","Data":"97a558d8edf0caa73c6220a5bcdcc685ebbe0256f158e160a0112e2fbba274b7"}
Sep 11 15:05:23.480369 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.480272    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf" event={"ID":"0cbc975c-be6f-4326-b8a1-b2de058f4086","Type":"ContainerDied","Data":"9af94bff9f260db806df606dc9a030d17b9b784e21e47058f3fa7f36bc869f11"}
Sep 11 15:05:23.480369 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.480304    2568 scope.go:117] "RemoveContainer" containerID="97a558d8edf0caa73c6220a5bcdcc685ebbe0256f158e160a0112e2fbba274b7"
Sep 11 15:05:23.494924 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.494895    2568 scope.go:117] "RemoveContainer" containerID="97a558d8edf0caa73c6220a5bcdcc685ebbe0256f158e160a0112e2fbba274b7"
Sep 11 15:05:23.495261 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:05:23.495241    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97a558d8edf0caa73c6220a5bcdcc685ebbe0256f158e160a0112e2fbba274b7\": container with ID starting with 97a558d8edf0caa73c6220a5bcdcc685ebbe0256f158e160a0112e2fbba274b7 not found: ID does not exist" containerID="97a558d8edf0caa73c6220a5bcdcc685ebbe0256f158e160a0112e2fbba274b7"
Sep 11 15:05:23.495344 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.495270    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97a558d8edf0caa73c6220a5bcdcc685ebbe0256f158e160a0112e2fbba274b7"} err="failed to get container status \"97a558d8edf0caa73c6220a5bcdcc685ebbe0256f158e160a0112e2fbba274b7\": rpc error: code = NotFound desc = could not find container \"97a558d8edf0caa73c6220a5bcdcc685ebbe0256f158e160a0112e2fbba274b7\": container with ID starting with 97a558d8edf0caa73c6220a5bcdcc685ebbe0256f158e160a0112e2fbba274b7 not found: ID does not exist"
Sep 11 15:05:23.512858 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.512594    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"]
Sep 11 15:05:23.518412 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.518375    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e7ca56-6c447b8464-rfxwf"]
Sep 11 15:05:23.559726 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.559697    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzlcz\" (UniqueName: \"kubernetes.io/projected/80136c17-90ca-4571-9164-f25620470160-kube-api-access-bzlcz\") pod \"80136c17-90ca-4571-9164-f25620470160\" (UID: \"80136c17-90ca-4571-9164-f25620470160\") "
Sep 11 15:05:23.559938 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.559831    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80136c17-90ca-4571-9164-f25620470160-plugins-config-volume\") pod \"80136c17-90ca-4571-9164-f25620470160\" (UID: \"80136c17-90ca-4571-9164-f25620470160\") "
Sep 11 15:05:23.560015 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.560003    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbc975c-be6f-4326-b8a1-b2de058f4086-metrics-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:05:23.560077 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.560017    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbsvf\" (UniqueName: \"kubernetes.io/projected/0cbc975c-be6f-4326-b8a1-b2de058f4086-kube-api-access-mbsvf\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:05:23.560077 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.560027    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbc975c-be6f-4326-b8a1-b2de058f4086-openshift-service-ca\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:05:23.560077 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.560035    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cbc975c-be6f-4326-b8a1-b2de058f4086-maas-api-tls\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:05:23.560239 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.560167    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80136c17-90ca-4571-9164-f25620470160-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "80136c17-90ca-4571-9164-f25620470160" (UID: "80136c17-90ca-4571-9164-f25620470160"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:05:23.561742 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.561715    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80136c17-90ca-4571-9164-f25620470160-kube-api-access-bzlcz" (OuterVolumeSpecName: "kube-api-access-bzlcz") pod "80136c17-90ca-4571-9164-f25620470160" (UID: "80136c17-90ca-4571-9164-f25620470160"). InnerVolumeSpecName "kube-api-access-bzlcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:05:23.660776 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.660693    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80136c17-90ca-4571-9164-f25620470160-plugins-config-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:05:23.660776 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:23.660732    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzlcz\" (UniqueName: \"kubernetes.io/projected/80136c17-90ca-4571-9164-f25620470160-kube-api-access-bzlcz\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:05:24.497836 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:24.497733    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv"
Sep 11 15:05:24.525062 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:24.525026    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv"]
Sep 11 15:05:24.531211 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:24.531184    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e7ca56-6954cfc9f7-ld4cv"]
Sep 11 15:05:24.663753 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:24.663721    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cbc975c-be6f-4326-b8a1-b2de058f4086" path="/var/lib/kubelet/pods/0cbc975c-be6f-4326-b8a1-b2de058f4086/volumes"
Sep 11 15:05:24.664132 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:24.664117    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80136c17-90ca-4571-9164-f25620470160" path="/var/lib/kubelet/pods/80136c17-90ca-4571-9164-f25620470160/volumes"
Sep 11 15:05:25.504713 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:25.504680    2568 generic.go:356] "Generic (PLEG): container finished" podID="6e993b0e-b03c-4eca-b18b-91356ca73783" containerID="67a35fe9b1c1c931673ee8821faa0646ab573bd00f8253bda85bf5a3f49b82f2" exitCode=0
Sep 11 15:05:25.505190 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:25.504747    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m" event={"ID":"6e993b0e-b03c-4eca-b18b-91356ca73783","Type":"ContainerDied","Data":"67a35fe9b1c1c931673ee8821faa0646ab573bd00f8253bda85bf5a3f49b82f2"}
Sep 11 15:05:25.505257 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:25.505245    2568 scope.go:117] "RemoveContainer" containerID="67a35fe9b1c1c931673ee8821faa0646ab573bd00f8253bda85bf5a3f49b82f2"
Sep 11 15:05:25.506358 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:25.506330    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:05:25.506446 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:25.506395    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:05:25.506486 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:25.506443    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:05:26.512616 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:26.512569    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m" event={"ID":"6e993b0e-b03c-4eca-b18b-91356ca73783","Type":"ContainerStarted","Data":"5a6f46ca3ed33e1bf106455d77f42823a3b97c3eb6eca1b19edb05dac76a9af3"}
Sep 11 15:05:26.845191 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:26.845152    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2" podUID="cb807d22-6c25-4b59-99b5-b79df420519c" containerName="main" containerID="cri-o://72bddf920e40c72e067c4897cf791f4813cfa16a4975b55c2a0bc3065dcef01e" gracePeriod=45
Sep 11 15:05:26.942983 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:26.942945    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:05:26.944028 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:26.944000    2568 patch_prober.go:28] interesting pod/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 11 15:05:26.944099 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:26.944057    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m" podUID="6e993b0e-b03c-4eca-b18b-91356ca73783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 11 15:05:27.156051 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.155274    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:05:27.196869 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.196834    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb807d22-6c25-4b59-99b5-b79df420519c-tls-certs\") pod \"cb807d22-6c25-4b59-99b5-b79df420519c\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") "
Sep 11 15:05:27.197055 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.196922    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq8jl\" (UniqueName: \"kubernetes.io/projected/cb807d22-6c25-4b59-99b5-b79df420519c-kube-api-access-sq8jl\") pod \"cb807d22-6c25-4b59-99b5-b79df420519c\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") "
Sep 11 15:05:27.197055 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.197018    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-model-cache\") pod \"cb807d22-6c25-4b59-99b5-b79df420519c\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") "
Sep 11 15:05:27.197180 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.197072    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-dshm\") pod \"cb807d22-6c25-4b59-99b5-b79df420519c\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") "
Sep 11 15:05:27.197180 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.197118    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-home\") pod \"cb807d22-6c25-4b59-99b5-b79df420519c\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") "
Sep 11 15:05:27.197180 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.197158    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-tmp-dir\") pod \"cb807d22-6c25-4b59-99b5-b79df420519c\" (UID: \"cb807d22-6c25-4b59-99b5-b79df420519c\") "
Sep 11 15:05:27.197598 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.197448    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-model-cache" (OuterVolumeSpecName: "model-cache") pod "cb807d22-6c25-4b59-99b5-b79df420519c" (UID: "cb807d22-6c25-4b59-99b5-b79df420519c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:05:27.198074 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.198046    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cb807d22-6c25-4b59-99b5-b79df420519c" (UID: "cb807d22-6c25-4b59-99b5-b79df420519c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:05:27.198356 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.198324    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-home" (OuterVolumeSpecName: "home") pod "cb807d22-6c25-4b59-99b5-b79df420519c" (UID: "cb807d22-6c25-4b59-99b5-b79df420519c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:05:27.199862 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.199836    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb807d22-6c25-4b59-99b5-b79df420519c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb807d22-6c25-4b59-99b5-b79df420519c" (UID: "cb807d22-6c25-4b59-99b5-b79df420519c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:05:27.200401 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.200367    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb807d22-6c25-4b59-99b5-b79df420519c-kube-api-access-sq8jl" (OuterVolumeSpecName: "kube-api-access-sq8jl") pod "cb807d22-6c25-4b59-99b5-b79df420519c" (UID: "cb807d22-6c25-4b59-99b5-b79df420519c"). InnerVolumeSpecName "kube-api-access-sq8jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:05:27.200502 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.200481    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-dshm" (OuterVolumeSpecName: "dshm") pod "cb807d22-6c25-4b59-99b5-b79df420519c" (UID: "cb807d22-6c25-4b59-99b5-b79df420519c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:05:27.298615 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.298586    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb807d22-6c25-4b59-99b5-b79df420519c-tls-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.298615 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.298613    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq8jl\" (UniqueName: \"kubernetes.io/projected/cb807d22-6c25-4b59-99b5-b79df420519c-kube-api-access-sq8jl\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.298615 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.298623    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-model-cache\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.298866 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.298631    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-dshm\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.298866 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.298639    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-home\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.298866 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.298648    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb807d22-6c25-4b59-99b5-b79df420519c-tmp-dir\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:05:27.530063 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.530023    2568 generic.go:356] "Generic (PLEG): container finished" podID="cb807d22-6c25-4b59-99b5-b79df420519c" containerID="72bddf920e40c72e067c4897cf791f4813cfa16a4975b55c2a0bc3065dcef01e" exitCode=0
Sep 11 15:05:27.530882 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.530860    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"
Sep 11 15:05:27.534061 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.534026    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2" event={"ID":"cb807d22-6c25-4b59-99b5-b79df420519c","Type":"ContainerDied","Data":"72bddf920e40c72e067c4897cf791f4813cfa16a4975b55c2a0bc3065dcef01e"}
Sep 11 15:05:27.534172 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.534079    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2" event={"ID":"cb807d22-6c25-4b59-99b5-b79df420519c","Type":"ContainerDied","Data":"282f4b67732f4ae93442c2f668efab8cf26108b73c9b4b8a992e003a1cad1349"}
Sep 11 15:05:27.534172 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.534105    2568 scope.go:117] "RemoveContainer" containerID="72bddf920e40c72e067c4897cf791f4813cfa16a4975b55c2a0bc3065dcef01e"
Sep 11 15:05:27.548887 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.548869    2568 scope.go:117] "RemoveContainer" containerID="72bddf920e40c72e067c4897cf791f4813cfa16a4975b55c2a0bc3065dcef01e"
Sep 11 15:05:27.549141 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:05:27.549120    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72bddf920e40c72e067c4897cf791f4813cfa16a4975b55c2a0bc3065dcef01e\": container with ID starting with 72bddf920e40c72e067c4897cf791f4813cfa16a4975b55c2a0bc3065dcef01e not found: ID does not exist" containerID="72bddf920e40c72e067c4897cf791f4813cfa16a4975b55c2a0bc3065dcef01e"
Sep 11 15:05:27.549184 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.549151    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72bddf920e40c72e067c4897cf791f4813cfa16a4975b55c2a0bc3065dcef01e"} err="failed to get container status \"72bddf920e40c72e067c4897cf791f4813cfa16a4975b55c2a0bc3065dcef01e\": rpc error: code = NotFound desc = could not find container \"72bddf920e40c72e067c4897cf791f4813cfa16a4975b55c2a0bc3065dcef01e\": container with ID starting with 72bddf920e40c72e067c4897cf791f4813cfa16a4975b55c2a0bc3065dcef01e not found: ID does not exist"
Sep 11 15:05:27.559416 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.559032    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"]
Sep 11 15:05:27.564528 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.564498    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e7ca56/embedding-w0-e7ca56-kserve-7c4987db85-2gbq2"]
Sep 11 15:05:27.943860 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.943745    2568 patch_prober.go:28] interesting pod/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 11 15:05:27.943860 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:27.943801    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m" podUID="6e993b0e-b03c-4eca-b18b-91356ca73783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 11 15:05:28.309191 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:28.309151    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:05:28.662802 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:28.662702    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb807d22-6c25-4b59-99b5-b79df420519c" path="/var/lib/kubelet/pods/cb807d22-6c25-4b59-99b5-b79df420519c/volumes"
Sep 11 15:05:28.943781 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:28.943701    2568 patch_prober.go:28] interesting pod/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 11 15:05:28.943781 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:28.943757    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m" podUID="6e993b0e-b03c-4eca-b18b-91356ca73783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 11 15:05:29.944047 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:29.944006    2568 patch_prober.go:28] interesting pod/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 11 15:05:29.944433 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:29.944069    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m" podUID="6e993b0e-b03c-4eca-b18b-91356ca73783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 11 15:05:30.256615 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.255645    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"]
Sep 11 15:05:30.256615 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.256451    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80136c17-90ca-4571-9164-f25620470160" containerName="payload-processing"
Sep 11 15:05:30.256615 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.256467    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="80136c17-90ca-4571-9164-f25620470160" containerName="payload-processing"
Sep 11 15:05:30.256615 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.256489    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cbc975c-be6f-4326-b8a1-b2de058f4086" containerName="maas-api"
Sep 11 15:05:30.256615 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.256498    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cbc975c-be6f-4326-b8a1-b2de058f4086" containerName="maas-api"
Sep 11 15:05:30.256615 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.256535    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb807d22-6c25-4b59-99b5-b79df420519c" containerName="main"
Sep 11 15:05:30.256615 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.256544    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb807d22-6c25-4b59-99b5-b79df420519c" containerName="main"
Sep 11 15:05:30.257072 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.256640    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cbc975c-be6f-4326-b8a1-b2de058f4086" containerName="maas-api"
Sep 11 15:05:30.257072 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.256651    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="80136c17-90ca-4571-9164-f25620470160" containerName="payload-processing"
Sep 11 15:05:30.257072 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.256676    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb807d22-6c25-4b59-99b5-b79df420519c" containerName="main"
Sep 11 15:05:30.261778 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.261749    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.265587 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.265556    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-14a808-openshift-default-dockercfg-tpr6r\""
Sep 11 15:05:30.268906 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.268877    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"]
Sep 11 15:05:30.352839 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.352285    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65a3141-53e6-4feb-8628-882a06453fdd-istio-token\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.352839 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.352338    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-workload-certs\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.352839 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.352420    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-istio-envoy\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.352839 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.352438    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-istio-data\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.352839 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.352476    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-workload-socket\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.352839 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.352503    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk4vc\" (UniqueName: \"kubernetes.io/projected/b65a3141-53e6-4feb-8628-882a06453fdd-kube-api-access-jk4vc\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.352839 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.352554    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65a3141-53e6-4feb-8628-882a06453fdd-istiod-ca-cert\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.352839 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.352578    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-credential-socket\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.352839 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.352600    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65a3141-53e6-4feb-8628-882a06453fdd-istio-podinfo\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.454018 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.453983    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65a3141-53e6-4feb-8628-882a06453fdd-istiod-ca-cert\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.454018 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.454021    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-credential-socket\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.454313 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.454037    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65a3141-53e6-4feb-8628-882a06453fdd-istio-podinfo\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.454313 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.454062    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65a3141-53e6-4feb-8628-882a06453fdd-istio-token\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.454313 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.454085    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-workload-certs\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.454313 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.454173    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-istio-envoy\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.454313 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.454198    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-istio-data\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.454313 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.454253    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-workload-socket\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.454313 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.454290    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk4vc\" (UniqueName: \"kubernetes.io/projected/b65a3141-53e6-4feb-8628-882a06453fdd-kube-api-access-jk4vc\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.454704 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.454522    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-credential-socket\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.454704 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.454571    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-workload-certs\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.454844 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.454789    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-istio-data\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.454905 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.454855    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-workload-socket\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.454905 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.454878    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65a3141-53e6-4feb-8628-882a06453fdd-istiod-ca-cert\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.456550 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.456531    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-istio-envoy\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.456860 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.456841    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65a3141-53e6-4feb-8628-882a06453fdd-istio-podinfo\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.463198 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.463175    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk4vc\" (UniqueName: \"kubernetes.io/projected/b65a3141-53e6-4feb-8628-882a06453fdd-kube-api-access-jk4vc\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.463379 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.463362    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65a3141-53e6-4feb-8628-882a06453fdd-istio-token\") pod \"e2e-api-b-14a808-openshift-default-767dc55cdb-52h28\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") " pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.575427 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.575386    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:30.882791 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:05:30.882761    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65a3141_53e6_4feb_8628_882a06453fdd.slice/crio-fa199199c1f0d52b835fa4c9db8d1768f05c326e87c208e144132403e630be2f WatchSource:0}: Error finding container fa199199c1f0d52b835fa4c9db8d1768f05c326e87c208e144132403e630be2f: Status 404 returned error can't find the container with id fa199199c1f0d52b835fa4c9db8d1768f05c326e87c208e144132403e630be2f
Sep 11 15:05:30.884428 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.884402    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"]
Sep 11 15:05:30.888474 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.888437    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:05:30.888550 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.888508    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:05:30.888550 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.888538    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:05:30.943360 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.943075    2568 patch_prober.go:28] interesting pod/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 11 15:05:30.943360 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:30.943158    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m" podUID="6e993b0e-b03c-4eca-b18b-91356ca73783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 11 15:05:31.563255 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:31.563212    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28" event={"ID":"b65a3141-53e6-4feb-8628-882a06453fdd","Type":"ContainerStarted","Data":"7998205db540b48aa8fe1252653fa3ad6fb9c15fdff552e15d20a36107a982fe"}
Sep 11 15:05:31.563766 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:31.563272    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28" event={"ID":"b65a3141-53e6-4feb-8628-882a06453fdd","Type":"ContainerStarted","Data":"fa199199c1f0d52b835fa4c9db8d1768f05c326e87c208e144132403e630be2f"}
Sep 11 15:05:31.575895 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:31.575866    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:31.577647 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:31.577622    2568 patch_prober.go:28] interesting pod/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 11 15:05:31.577769 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:31.577671    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28" podUID="b65a3141-53e6-4feb-8628-882a06453fdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 11 15:05:31.592722 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:31.592666    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28" podStartSLOduration=1.592646316 podStartE2EDuration="1.592646316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:05:31.588600149 +0000 UTC m=+1511.376864448" watchObservedRunningTime="2026-09-11 15:05:31.592646316 +0000 UTC m=+1511.380910617"
Sep 11 15:05:31.947829 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:31.947730    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:05:31.948862 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:31.948840    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:05:32.579956 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:32.579924    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:33.578679 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:33.578645    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:33.579544 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:33.579523    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:05:39.502230 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.502192    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5"]
Sep 11 15:05:39.507207 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.507187    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5"
Sep 11 15:05:39.509951 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.509929    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-14a808-dockercfg-kg5lm\""
Sep 11 15:05:39.510707 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.510691    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-14a808\""
Sep 11 15:05:39.516670 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.516604    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5"]
Sep 11 15:05:39.520727 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.520703    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r"]
Sep 11 15:05:39.524346 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.524330    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r"
Sep 11 15:05:39.536256 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.536235    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r"]
Sep 11 15:05:39.655876 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.655831    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcqgt\" (UniqueName: \"kubernetes.io/projected/117a2b14-548b-464e-ad83-8f491cf41057-kube-api-access-xcqgt\") pod \"payload-processing-e2e-api-b-14a808-7b686594df-tvg2r\" (UID: \"117a2b14-548b-464e-ad83-8f491cf41057\") " pod="openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r"
Sep 11 15:05:39.656064 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.655906    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9880a59b-7639-4db0-9760-1b50f3144b59-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5\" (UID: \"9880a59b-7639-4db0-9760-1b50f3144b59\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5"
Sep 11 15:05:39.656064 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.656034    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/117a2b14-548b-464e-ad83-8f491cf41057-plugins-config-volume\") pod \"payload-processing-e2e-api-b-14a808-7b686594df-tvg2r\" (UID: \"117a2b14-548b-464e-ad83-8f491cf41057\") " pod="openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r"
Sep 11 15:05:39.656179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.656069    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smsj8\" (UniqueName: \"kubernetes.io/projected/9880a59b-7639-4db0-9760-1b50f3144b59-kube-api-access-smsj8\") pod \"payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5\" (UID: \"9880a59b-7639-4db0-9760-1b50f3144b59\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5"
Sep 11 15:05:39.756950 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.756863    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/117a2b14-548b-464e-ad83-8f491cf41057-plugins-config-volume\") pod \"payload-processing-e2e-api-b-14a808-7b686594df-tvg2r\" (UID: \"117a2b14-548b-464e-ad83-8f491cf41057\") " pod="openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r"
Sep 11 15:05:39.756950 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.756910    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smsj8\" (UniqueName: \"kubernetes.io/projected/9880a59b-7639-4db0-9760-1b50f3144b59-kube-api-access-smsj8\") pod \"payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5\" (UID: \"9880a59b-7639-4db0-9760-1b50f3144b59\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5"
Sep 11 15:05:39.757186 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.757085    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcqgt\" (UniqueName: \"kubernetes.io/projected/117a2b14-548b-464e-ad83-8f491cf41057-kube-api-access-xcqgt\") pod \"payload-processing-e2e-api-b-14a808-7b686594df-tvg2r\" (UID: \"117a2b14-548b-464e-ad83-8f491cf41057\") " pod="openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r"
Sep 11 15:05:39.757186 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.757173    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9880a59b-7639-4db0-9760-1b50f3144b59-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5\" (UID: \"9880a59b-7639-4db0-9760-1b50f3144b59\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5"
Sep 11 15:05:39.757658 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.757635    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/117a2b14-548b-464e-ad83-8f491cf41057-plugins-config-volume\") pod \"payload-processing-e2e-api-b-14a808-7b686594df-tvg2r\" (UID: \"117a2b14-548b-464e-ad83-8f491cf41057\") " pod="openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r"
Sep 11 15:05:39.757946 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.757926    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9880a59b-7639-4db0-9760-1b50f3144b59-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5\" (UID: \"9880a59b-7639-4db0-9760-1b50f3144b59\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5"
Sep 11 15:05:39.766084 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.766062    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcqgt\" (UniqueName: \"kubernetes.io/projected/117a2b14-548b-464e-ad83-8f491cf41057-kube-api-access-xcqgt\") pod \"payload-processing-e2e-api-b-14a808-7b686594df-tvg2r\" (UID: \"117a2b14-548b-464e-ad83-8f491cf41057\") " pod="openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r"
Sep 11 15:05:39.766162 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.766127    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smsj8\" (UniqueName: \"kubernetes.io/projected/9880a59b-7639-4db0-9760-1b50f3144b59-kube-api-access-smsj8\") pod \"payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5\" (UID: \"9880a59b-7639-4db0-9760-1b50f3144b59\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5"
Sep 11 15:05:39.822058 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.822022    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5"
Sep 11 15:05:39.835785 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:39.835758    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r"
Sep 11 15:05:40.116878 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:40.116846    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5"]
Sep 11 15:05:40.118335 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:05:40.118305    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9880a59b_7639_4db0_9760_1b50f3144b59.slice/crio-2dbe401901f5819aa31f366252b7ecba3f01e61d4aa2d0fb9a21683ffb45a146 WatchSource:0}: Error finding container 2dbe401901f5819aa31f366252b7ecba3f01e61d4aa2d0fb9a21683ffb45a146: Status 404 returned error can't find the container with id 2dbe401901f5819aa31f366252b7ecba3f01e61d4aa2d0fb9a21683ffb45a146
Sep 11 15:05:40.143950 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:40.143924    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r"]
Sep 11 15:05:40.156244 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:05:40.156216    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117a2b14_548b_464e_ad83_8f491cf41057.slice/crio-a4af4c48d02a99133345ed233c9647ee15e2cfd96d1d21ade5782e2538c1b142 WatchSource:0}: Error finding container a4af4c48d02a99133345ed233c9647ee15e2cfd96d1d21ade5782e2538c1b142: Status 404 returned error can't find the container with id a4af4c48d02a99133345ed233c9647ee15e2cfd96d1d21ade5782e2538c1b142
Sep 11 15:05:40.620184 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:40.620143    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r" event={"ID":"117a2b14-548b-464e-ad83-8f491cf41057","Type":"ContainerStarted","Data":"cc2ce55c6ee48c37a9a87ae825ea3fda0e648a9773d715a5f85fca9e55e0cf6f"}
Sep 11 15:05:40.620184 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:40.620194    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r" event={"ID":"117a2b14-548b-464e-ad83-8f491cf41057","Type":"ContainerStarted","Data":"a4af4c48d02a99133345ed233c9647ee15e2cfd96d1d21ade5782e2538c1b142"}
Sep 11 15:05:40.620728 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:40.620214    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r"
Sep 11 15:05:40.622270 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:40.622240    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5" event={"ID":"9880a59b-7639-4db0-9760-1b50f3144b59","Type":"ContainerStarted","Data":"0de97b115799f33a024f63fb7d24ffd3476151fe5bd269334384d7fa6d6f802a"}
Sep 11 15:05:40.622415 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:40.622292    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5" event={"ID":"9880a59b-7639-4db0-9760-1b50f3144b59","Type":"ContainerStarted","Data":"2dbe401901f5819aa31f366252b7ecba3f01e61d4aa2d0fb9a21683ffb45a146"}
Sep 11 15:05:40.622415 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:40.622321    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5"
Sep 11 15:05:40.639880 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:40.639832    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r" podStartSLOduration=1.639802012 podStartE2EDuration="1.639802012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:05:40.637394059 +0000 UTC m=+1520.425658383" watchObservedRunningTime="2026-09-11 15:05:40.639802012 +0000 UTC m=+1520.428066310"
Sep 11 15:05:40.655901 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:40.655852    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5" podStartSLOduration=1.655838739 podStartE2EDuration="1.655838739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:05:40.653861558 +0000 UTC m=+1520.442125858" watchObservedRunningTime="2026-09-11 15:05:40.655838739 +0000 UTC m=+1520.444103039"
Sep 11 15:05:51.630062 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:51.629973    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5"
Sep 11 15:05:51.630562 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:05:51.630264    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r"
Sep 11 15:06:09.790417 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:09.790385    2568 generic.go:356] "Generic (PLEG): container finished" podID="b65a3141-53e6-4feb-8628-882a06453fdd" containerID="7998205db540b48aa8fe1252653fa3ad6fb9c15fdff552e15d20a36107a982fe" exitCode=0
Sep 11 15:06:09.790849 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:09.790466    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28" event={"ID":"b65a3141-53e6-4feb-8628-882a06453fdd","Type":"ContainerDied","Data":"7998205db540b48aa8fe1252653fa3ad6fb9c15fdff552e15d20a36107a982fe"}
Sep 11 15:06:09.790930 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:09.790914    2568 scope.go:117] "RemoveContainer" containerID="7998205db540b48aa8fe1252653fa3ad6fb9c15fdff552e15d20a36107a982fe"
Sep 11 15:06:09.792087 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:09.792058    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:06:09.792148 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:09.792127    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:06:09.792192 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:09.792157    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:06:10.801371 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:10.801336    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28" event={"ID":"b65a3141-53e6-4feb-8628-882a06453fdd","Type":"ContainerStarted","Data":"ebe7d95d9a51eab692636a6fff405b44c98f2fed7b7357eb5f5203f9e40098e3"}
Sep 11 15:06:11.575599 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:11.575564    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:06:11.575872 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:11.575850    2568 patch_prober.go:28] interesting pod/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 11 15:06:11.575932 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:11.575913    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28" podUID="b65a3141-53e6-4feb-8628-882a06453fdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 11 15:06:12.576180 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:12.576143    2568 patch_prober.go:28] interesting pod/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 11 15:06:12.576564 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:12.576212    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28" podUID="b65a3141-53e6-4feb-8628-882a06453fdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 11 15:06:13.575960 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:13.575919    2568 patch_prober.go:28] interesting pod/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 11 15:06:13.576139 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:13.575979    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28" podUID="b65a3141-53e6-4feb-8628-882a06453fdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 11 15:06:14.576502 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:14.576467    2568 patch_prober.go:28] interesting pod/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 11 15:06:14.576912 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:14.576525    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28" podUID="b65a3141-53e6-4feb-8628-882a06453fdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 11 15:06:15.576288 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:15.576244    2568 patch_prober.go:28] interesting pod/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 11 15:06:15.576468 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:15.576310    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28" podUID="b65a3141-53e6-4feb-8628-882a06453fdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 11 15:06:16.580526 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:16.580489    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:06:16.580948 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:16.580849    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:06:16.581530 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:16.581509    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:06:20.807575 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:20.807538    2568 scope.go:117] "RemoveContainer" containerID="4f0945ce85084fa08c576dde6f3952eaec5e417740775d14b8830318864c7305"
Sep 11 15:06:22.805255 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:22.805216    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf"]
Sep 11 15:06:22.805653 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:22.805443    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf" podUID="7264546b-16e5-4b54-9542-5943919be700" containerName="payload-pre-processing" containerID="cri-o://028765c7cb28f5850004693430f77fd1b9c4991ef9aa9757d9d1fa0b5e76bdc8" gracePeriod=30
Sep 11 15:06:23.108238 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:23.108211    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf"
Sep 11 15:06:23.144256 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:23.141110    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4z7cp\" (UniqueName: \"kubernetes.io/projected/7264546b-16e5-4b54-9542-5943919be700-kube-api-access-4z7cp\") pod \"7264546b-16e5-4b54-9542-5943919be700\" (UID: \"7264546b-16e5-4b54-9542-5943919be700\") "
Sep 11 15:06:23.144256 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:23.141185    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7264546b-16e5-4b54-9542-5943919be700-plugins-config-volume\") pod \"7264546b-16e5-4b54-9542-5943919be700\" (UID: \"7264546b-16e5-4b54-9542-5943919be700\") "
Sep 11 15:06:23.144256 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:23.141842    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7264546b-16e5-4b54-9542-5943919be700-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7264546b-16e5-4b54-9542-5943919be700" (UID: "7264546b-16e5-4b54-9542-5943919be700"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:06:23.144706 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:23.144681    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7264546b-16e5-4b54-9542-5943919be700-kube-api-access-4z7cp" (OuterVolumeSpecName: "kube-api-access-4z7cp") pod "7264546b-16e5-4b54-9542-5943919be700" (UID: "7264546b-16e5-4b54-9542-5943919be700"). InnerVolumeSpecName "kube-api-access-4z7cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:06:23.242338 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:23.242301    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z7cp\" (UniqueName: \"kubernetes.io/projected/7264546b-16e5-4b54-9542-5943919be700-kube-api-access-4z7cp\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:23.242338 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:23.242333    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7264546b-16e5-4b54-9542-5943919be700-plugins-config-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:23.882054 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:23.882024    2568 generic.go:356] "Generic (PLEG): container finished" podID="7264546b-16e5-4b54-9542-5943919be700" containerID="028765c7cb28f5850004693430f77fd1b9c4991ef9aa9757d9d1fa0b5e76bdc8" exitCode=0
Sep 11 15:06:23.882509 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:23.882085    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf"
Sep 11 15:06:23.882509 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:23.882135    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf" event={"ID":"7264546b-16e5-4b54-9542-5943919be700","Type":"ContainerDied","Data":"028765c7cb28f5850004693430f77fd1b9c4991ef9aa9757d9d1fa0b5e76bdc8"}
Sep 11 15:06:23.882509 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:23.882186    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf" event={"ID":"7264546b-16e5-4b54-9542-5943919be700","Type":"ContainerDied","Data":"ad9e9c5deb6585ec2becf1fff44333f0dcd8b7ba6820d90f8ecc4711572e8ec9"}
Sep 11 15:06:23.882509 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:23.882208    2568 scope.go:117] "RemoveContainer" containerID="028765c7cb28f5850004693430f77fd1b9c4991ef9aa9757d9d1fa0b5e76bdc8"
Sep 11 15:06:23.892269 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:23.892249    2568 scope.go:117] "RemoveContainer" containerID="028765c7cb28f5850004693430f77fd1b9c4991ef9aa9757d9d1fa0b5e76bdc8"
Sep 11 15:06:23.892522 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:06:23.892505    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028765c7cb28f5850004693430f77fd1b9c4991ef9aa9757d9d1fa0b5e76bdc8\": container with ID starting with 028765c7cb28f5850004693430f77fd1b9c4991ef9aa9757d9d1fa0b5e76bdc8 not found: ID does not exist" containerID="028765c7cb28f5850004693430f77fd1b9c4991ef9aa9757d9d1fa0b5e76bdc8"
Sep 11 15:06:23.892561 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:23.892532    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028765c7cb28f5850004693430f77fd1b9c4991ef9aa9757d9d1fa0b5e76bdc8"} err="failed to get container status \"028765c7cb28f5850004693430f77fd1b9c4991ef9aa9757d9d1fa0b5e76bdc8\": rpc error: code = NotFound desc = could not find container \"028765c7cb28f5850004693430f77fd1b9c4991ef9aa9757d9d1fa0b5e76bdc8\": container with ID starting with 028765c7cb28f5850004693430f77fd1b9c4991ef9aa9757d9d1fa0b5e76bdc8 not found: ID does not exist"
Sep 11 15:06:23.904595 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:23.904572    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf"]
Sep 11 15:06:23.909831 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:23.909795    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-54b7f9-5b4bdccd54-bs9vf"]
Sep 11 15:06:24.662716 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:24.662680    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7264546b-16e5-4b54-9542-5943919be700" path="/var/lib/kubelet/pods/7264546b-16e5-4b54-9542-5943919be700/volumes"
Sep 11 15:06:30.188739 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:30.188705    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"]
Sep 11 15:06:30.189251 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:30.188924    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m" podUID="6e993b0e-b03c-4eca-b18b-91356ca73783" containerName="istio-proxy" containerID="cri-o://5a6f46ca3ed33e1bf106455d77f42823a3b97c3eb6eca1b19edb05dac76a9af3" gracePeriod=30
Sep 11 15:06:31.954598 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:31.954542    2568 patch_prober.go:28] interesting pod/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 15:06:31.955321 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:31.955286    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m" podUID="6e993b0e-b03c-4eca-b18b-91356ca73783" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 15:06:32.497527 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.497488    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"]
Sep 11 15:06:32.497799 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.497732    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4" podUID="cd30c533-7634-4d21-a6e0-249dbba1e43c" containerName="maas-api" containerID="cri-o://7c3c064ed651dbb53c815b389537c9ababf72d204f43898f45088a07673db73c" gracePeriod=30
Sep 11 15:06:32.561842 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.561795    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4"]
Sep 11 15:06:32.562070 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.562044    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4" podUID="dead80a8-6b34-4408-ac1b-867c2ec0b91f" containerName="payload-processing" containerID="cri-o://6c8965089c4cc0a1475c2cd7159c6a70596bef7a9e0ea7e1f9de594e22f283f1" gracePeriod=30
Sep 11 15:06:32.571050 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.570986    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n"]
Sep 11 15:06:32.571276 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.571223    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n" podUID="74e3498c-b895-4e07-82af-9586e0ef670d" containerName="payload-pre-processing" containerID="cri-o://1c86b7bea5734958e8abeed7cf63dba00d37e4a2ad03cbf3b35790ecfc91254c" gracePeriod=30
Sep 11 15:06:32.873025 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.872998    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4"
Sep 11 15:06:32.897391 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.897369    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n"
Sep 11 15:06:32.942704 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.941992    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dead80a8-6b34-4408-ac1b-867c2ec0b91f-plugins-config-volume\") pod \"dead80a8-6b34-4408-ac1b-867c2ec0b91f\" (UID: \"dead80a8-6b34-4408-ac1b-867c2ec0b91f\") "
Sep 11 15:06:32.942704 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.942121    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdj2b\" (UniqueName: \"kubernetes.io/projected/dead80a8-6b34-4408-ac1b-867c2ec0b91f-kube-api-access-xdj2b\") pod \"dead80a8-6b34-4408-ac1b-867c2ec0b91f\" (UID: \"dead80a8-6b34-4408-ac1b-867c2ec0b91f\") "
Sep 11 15:06:32.942704 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.942188    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqg2p\" (UniqueName: \"kubernetes.io/projected/74e3498c-b895-4e07-82af-9586e0ef670d-kube-api-access-dqg2p\") pod \"74e3498c-b895-4e07-82af-9586e0ef670d\" (UID: \"74e3498c-b895-4e07-82af-9586e0ef670d\") "
Sep 11 15:06:32.942704 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.942267    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e3498c-b895-4e07-82af-9586e0ef670d-plugins-config-volume\") pod \"74e3498c-b895-4e07-82af-9586e0ef670d\" (UID: \"74e3498c-b895-4e07-82af-9586e0ef670d\") "
Sep 11 15:06:32.942704 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.942367    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dead80a8-6b34-4408-ac1b-867c2ec0b91f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dead80a8-6b34-4408-ac1b-867c2ec0b91f" (UID: "dead80a8-6b34-4408-ac1b-867c2ec0b91f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:06:32.942704 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.942676    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e3498c-b895-4e07-82af-9586e0ef670d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74e3498c-b895-4e07-82af-9586e0ef670d" (UID: "74e3498c-b895-4e07-82af-9586e0ef670d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:06:32.943180 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.942757    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74e3498c-b895-4e07-82af-9586e0ef670d-plugins-config-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:32.943180 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.942771    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dead80a8-6b34-4408-ac1b-867c2ec0b91f-plugins-config-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:32.943912 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.943866    2568 generic.go:356] "Generic (PLEG): container finished" podID="dead80a8-6b34-4408-ac1b-867c2ec0b91f" containerID="6c8965089c4cc0a1475c2cd7159c6a70596bef7a9e0ea7e1f9de594e22f283f1" exitCode=0
Sep 11 15:06:32.944043 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.943936    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4"
Sep 11 15:06:32.944043 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.943997    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4" event={"ID":"dead80a8-6b34-4408-ac1b-867c2ec0b91f","Type":"ContainerDied","Data":"6c8965089c4cc0a1475c2cd7159c6a70596bef7a9e0ea7e1f9de594e22f283f1"}
Sep 11 15:06:32.944161 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.944067    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4" event={"ID":"dead80a8-6b34-4408-ac1b-867c2ec0b91f","Type":"ContainerDied","Data":"2b40f0b069b6d31cf21b5564adf8d2f1d68e78965575298760015d09e9882740"}
Sep 11 15:06:32.944161 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.944138    2568 scope.go:117] "RemoveContainer" containerID="6c8965089c4cc0a1475c2cd7159c6a70596bef7a9e0ea7e1f9de594e22f283f1"
Sep 11 15:06:32.944492 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.944430    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dead80a8-6b34-4408-ac1b-867c2ec0b91f-kube-api-access-xdj2b" (OuterVolumeSpecName: "kube-api-access-xdj2b") pod "dead80a8-6b34-4408-ac1b-867c2ec0b91f" (UID: "dead80a8-6b34-4408-ac1b-867c2ec0b91f"). InnerVolumeSpecName "kube-api-access-xdj2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:06:32.944867 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.944811    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e3498c-b895-4e07-82af-9586e0ef670d-kube-api-access-dqg2p" (OuterVolumeSpecName: "kube-api-access-dqg2p") pod "74e3498c-b895-4e07-82af-9586e0ef670d" (UID: "74e3498c-b895-4e07-82af-9586e0ef670d"). InnerVolumeSpecName "kube-api-access-dqg2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:06:32.946439 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.946420    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n"
Sep 11 15:06:32.946543 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.946465    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n" event={"ID":"74e3498c-b895-4e07-82af-9586e0ef670d","Type":"ContainerDied","Data":"1c86b7bea5734958e8abeed7cf63dba00d37e4a2ad03cbf3b35790ecfc91254c"}
Sep 11 15:06:32.946674 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.946647    2568 generic.go:356] "Generic (PLEG): container finished" podID="74e3498c-b895-4e07-82af-9586e0ef670d" containerID="1c86b7bea5734958e8abeed7cf63dba00d37e4a2ad03cbf3b35790ecfc91254c" exitCode=0
Sep 11 15:06:32.946802 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.946782    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n" event={"ID":"74e3498c-b895-4e07-82af-9586e0ef670d","Type":"ContainerDied","Data":"ab4d73604d67616e478dfefa8c147cc7e12647f5d8249f3cece0bf09844a3308"}
Sep 11 15:06:32.949456 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.949430    2568 generic.go:356] "Generic (PLEG): container finished" podID="cd30c533-7634-4d21-a6e0-249dbba1e43c" containerID="7c3c064ed651dbb53c815b389537c9ababf72d204f43898f45088a07673db73c" exitCode=0
Sep 11 15:06:32.949546 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.949506    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4" event={"ID":"cd30c533-7634-4d21-a6e0-249dbba1e43c","Type":"ContainerDied","Data":"7c3c064ed651dbb53c815b389537c9ababf72d204f43898f45088a07673db73c"}
Sep 11 15:06:32.955968 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.955851    2568 scope.go:117] "RemoveContainer" containerID="6c8965089c4cc0a1475c2cd7159c6a70596bef7a9e0ea7e1f9de594e22f283f1"
Sep 11 15:06:32.956246 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:06:32.956167    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c8965089c4cc0a1475c2cd7159c6a70596bef7a9e0ea7e1f9de594e22f283f1\": container with ID starting with 6c8965089c4cc0a1475c2cd7159c6a70596bef7a9e0ea7e1f9de594e22f283f1 not found: ID does not exist" containerID="6c8965089c4cc0a1475c2cd7159c6a70596bef7a9e0ea7e1f9de594e22f283f1"
Sep 11 15:06:32.956246 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.956213    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c8965089c4cc0a1475c2cd7159c6a70596bef7a9e0ea7e1f9de594e22f283f1"} err="failed to get container status \"6c8965089c4cc0a1475c2cd7159c6a70596bef7a9e0ea7e1f9de594e22f283f1\": rpc error: code = NotFound desc = could not find container \"6c8965089c4cc0a1475c2cd7159c6a70596bef7a9e0ea7e1f9de594e22f283f1\": container with ID starting with 6c8965089c4cc0a1475c2cd7159c6a70596bef7a9e0ea7e1f9de594e22f283f1 not found: ID does not exist"
Sep 11 15:06:32.956246 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.956235    2568 scope.go:117] "RemoveContainer" containerID="1c86b7bea5734958e8abeed7cf63dba00d37e4a2ad03cbf3b35790ecfc91254c"
Sep 11 15:06:32.966055 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.966035    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"
Sep 11 15:06:32.966331 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.966317    2568 scope.go:117] "RemoveContainer" containerID="1c86b7bea5734958e8abeed7cf63dba00d37e4a2ad03cbf3b35790ecfc91254c"
Sep 11 15:06:32.966581 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:06:32.966566    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c86b7bea5734958e8abeed7cf63dba00d37e4a2ad03cbf3b35790ecfc91254c\": container with ID starting with 1c86b7bea5734958e8abeed7cf63dba00d37e4a2ad03cbf3b35790ecfc91254c not found: ID does not exist" containerID="1c86b7bea5734958e8abeed7cf63dba00d37e4a2ad03cbf3b35790ecfc91254c"
Sep 11 15:06:32.966624 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.966586    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c86b7bea5734958e8abeed7cf63dba00d37e4a2ad03cbf3b35790ecfc91254c"} err="failed to get container status \"1c86b7bea5734958e8abeed7cf63dba00d37e4a2ad03cbf3b35790ecfc91254c\": rpc error: code = NotFound desc = could not find container \"1c86b7bea5734958e8abeed7cf63dba00d37e4a2ad03cbf3b35790ecfc91254c\": container with ID starting with 1c86b7bea5734958e8abeed7cf63dba00d37e4a2ad03cbf3b35790ecfc91254c not found: ID does not exist"
Sep 11 15:06:32.972525 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.972504    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n"]
Sep 11 15:06:32.977005 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:32.976983    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-837632-77765b66db-st47n"]
Sep 11 15:06:33.044046 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.043962    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc22q\" (UniqueName: \"kubernetes.io/projected/cd30c533-7634-4d21-a6e0-249dbba1e43c-kube-api-access-lc22q\") pod \"cd30c533-7634-4d21-a6e0-249dbba1e43c\" (UID: \"cd30c533-7634-4d21-a6e0-249dbba1e43c\") "
Sep 11 15:06:33.044046 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.044021    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd30c533-7634-4d21-a6e0-249dbba1e43c-openshift-service-ca\") pod \"cd30c533-7634-4d21-a6e0-249dbba1e43c\" (UID: \"cd30c533-7634-4d21-a6e0-249dbba1e43c\") "
Sep 11 15:06:33.044324 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.044058    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd30c533-7634-4d21-a6e0-249dbba1e43c-metrics-certs\") pod \"cd30c533-7634-4d21-a6e0-249dbba1e43c\" (UID: \"cd30c533-7634-4d21-a6e0-249dbba1e43c\") "
Sep 11 15:06:33.044324 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.044130    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd30c533-7634-4d21-a6e0-249dbba1e43c-maas-api-tls\") pod \"cd30c533-7634-4d21-a6e0-249dbba1e43c\" (UID: \"cd30c533-7634-4d21-a6e0-249dbba1e43c\") "
Sep 11 15:06:33.044434 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.044412    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqg2p\" (UniqueName: \"kubernetes.io/projected/74e3498c-b895-4e07-82af-9586e0ef670d-kube-api-access-dqg2p\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:33.044492 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.044427    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd30c533-7634-4d21-a6e0-249dbba1e43c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd30c533-7634-4d21-a6e0-249dbba1e43c" (UID: "cd30c533-7634-4d21-a6e0-249dbba1e43c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:06:33.044492 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.044438    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdj2b\" (UniqueName: \"kubernetes.io/projected/dead80a8-6b34-4408-ac1b-867c2ec0b91f-kube-api-access-xdj2b\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:33.046202 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.046177    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd30c533-7634-4d21-a6e0-249dbba1e43c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd30c533-7634-4d21-a6e0-249dbba1e43c" (UID: "cd30c533-7634-4d21-a6e0-249dbba1e43c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:06:33.046202 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.046196    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd30c533-7634-4d21-a6e0-249dbba1e43c-kube-api-access-lc22q" (OuterVolumeSpecName: "kube-api-access-lc22q") pod "cd30c533-7634-4d21-a6e0-249dbba1e43c" (UID: "cd30c533-7634-4d21-a6e0-249dbba1e43c"). InnerVolumeSpecName "kube-api-access-lc22q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:06:33.046327 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.046213    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd30c533-7634-4d21-a6e0-249dbba1e43c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cd30c533-7634-4d21-a6e0-249dbba1e43c" (UID: "cd30c533-7634-4d21-a6e0-249dbba1e43c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:06:33.148755 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.148673    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd30c533-7634-4d21-a6e0-249dbba1e43c-maas-api-tls\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:33.148755 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.148719    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc22q\" (UniqueName: \"kubernetes.io/projected/cd30c533-7634-4d21-a6e0-249dbba1e43c-kube-api-access-lc22q\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:33.148755 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.148735    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd30c533-7634-4d21-a6e0-249dbba1e43c-openshift-service-ca\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:33.148755 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.148749    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd30c533-7634-4d21-a6e0-249dbba1e43c-metrics-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:33.267203 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.267170    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4"]
Sep 11 15:06:33.269318 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.269297    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-837632-5fddcb76c7-njvt4"]
Sep 11 15:06:33.958155 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.958067    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"
Sep 11 15:06:33.958155 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.958113    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4" event={"ID":"cd30c533-7634-4d21-a6e0-249dbba1e43c","Type":"ContainerDied","Data":"3481019ee1d6426129c570de979fa3d5cf3c743f49f583c2ea35c21960eed6b3"}
Sep 11 15:06:33.958658 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.958182    2568 scope.go:117] "RemoveContainer" containerID="7c3c064ed651dbb53c815b389537c9ababf72d204f43898f45088a07673db73c"
Sep 11 15:06:33.984846 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.984801    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"]
Sep 11 15:06:33.990542 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:33.990518    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-837632-cc6bfc846-k7jb4"]
Sep 11 15:06:34.667572 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:34.667535    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74e3498c-b895-4e07-82af-9586e0ef670d" path="/var/lib/kubelet/pods/74e3498c-b895-4e07-82af-9586e0ef670d/volumes"
Sep 11 15:06:34.667955 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:34.667940    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd30c533-7634-4d21-a6e0-249dbba1e43c" path="/var/lib/kubelet/pods/cd30c533-7634-4d21-a6e0-249dbba1e43c/volumes"
Sep 11 15:06:34.668460 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:34.668442    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dead80a8-6b34-4408-ac1b-867c2ec0b91f" path="/var/lib/kubelet/pods/dead80a8-6b34-4408-ac1b-867c2ec0b91f/volumes"
Sep 11 15:06:35.461934 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.461837    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:06:35.573210 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.573176    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-workload-certs\") pod \"6e993b0e-b03c-4eca-b18b-91356ca73783\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") "
Sep 11 15:06:35.573210 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.573211    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-envoy\") pod \"6e993b0e-b03c-4eca-b18b-91356ca73783\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") "
Sep 11 15:06:35.573427 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.573236    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-token\") pod \"6e993b0e-b03c-4eca-b18b-91356ca73783\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") "
Sep 11 15:06:35.573427 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.573273    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-credential-socket\") pod \"6e993b0e-b03c-4eca-b18b-91356ca73783\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") "
Sep 11 15:06:35.573427 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.573395    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-podinfo\") pod \"6e993b0e-b03c-4eca-b18b-91356ca73783\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") "
Sep 11 15:06:35.573615 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.573427    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-data\") pod \"6e993b0e-b03c-4eca-b18b-91356ca73783\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") "
Sep 11 15:06:35.573615 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.573469    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e993b0e-b03c-4eca-b18b-91356ca73783-istiod-ca-cert\") pod \"6e993b0e-b03c-4eca-b18b-91356ca73783\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") "
Sep 11 15:06:35.573615 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.573515    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6e993b0e-b03c-4eca-b18b-91356ca73783" (UID: "6e993b0e-b03c-4eca-b18b-91356ca73783"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:06:35.573615 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.573554    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86s7d\" (UniqueName: \"kubernetes.io/projected/6e993b0e-b03c-4eca-b18b-91356ca73783-kube-api-access-86s7d\") pod \"6e993b0e-b03c-4eca-b18b-91356ca73783\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") "
Sep 11 15:06:35.573871 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.573630    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-workload-socket\") pod \"6e993b0e-b03c-4eca-b18b-91356ca73783\" (UID: \"6e993b0e-b03c-4eca-b18b-91356ca73783\") "
Sep 11 15:06:35.573871 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.573670    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6e993b0e-b03c-4eca-b18b-91356ca73783" (UID: "6e993b0e-b03c-4eca-b18b-91356ca73783"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:06:35.573871 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.573721    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-data" (OuterVolumeSpecName: "istio-data") pod "6e993b0e-b03c-4eca-b18b-91356ca73783" (UID: "6e993b0e-b03c-4eca-b18b-91356ca73783"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:06:35.574043 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.573897    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e993b0e-b03c-4eca-b18b-91356ca73783-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6e993b0e-b03c-4eca-b18b-91356ca73783" (UID: "6e993b0e-b03c-4eca-b18b-91356ca73783"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:06:35.574043 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.574027    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6e993b0e-b03c-4eca-b18b-91356ca73783" (UID: "6e993b0e-b03c-4eca-b18b-91356ca73783"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:06:35.574167 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.574138    2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-data\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:35.574235 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.574171    2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e993b0e-b03c-4eca-b18b-91356ca73783-istiod-ca-cert\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:35.574235 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.574180    2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-workload-socket\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:35.574235 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.574190    2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-workload-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:35.574235 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.574208    2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-credential-socket\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:35.575568 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.575544    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6e993b0e-b03c-4eca-b18b-91356ca73783" (UID: "6e993b0e-b03c-4eca-b18b-91356ca73783"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 15:06:35.575706 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.575683    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-token" (OuterVolumeSpecName: "istio-token") pod "6e993b0e-b03c-4eca-b18b-91356ca73783" (UID: "6e993b0e-b03c-4eca-b18b-91356ca73783"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:06:35.575910 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.575888    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6e993b0e-b03c-4eca-b18b-91356ca73783" (UID: "6e993b0e-b03c-4eca-b18b-91356ca73783"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:06:35.575993 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.575903    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e993b0e-b03c-4eca-b18b-91356ca73783-kube-api-access-86s7d" (OuterVolumeSpecName: "kube-api-access-86s7d") pod "6e993b0e-b03c-4eca-b18b-91356ca73783" (UID: "6e993b0e-b03c-4eca-b18b-91356ca73783"). InnerVolumeSpecName "kube-api-access-86s7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:06:35.674831 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.674787    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86s7d\" (UniqueName: \"kubernetes.io/projected/6e993b0e-b03c-4eca-b18b-91356ca73783-kube-api-access-86s7d\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:35.674831 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.674834    2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-envoy\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:35.675016 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.674863    2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-token\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:35.675016 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.674871    2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e993b0e-b03c-4eca-b18b-91356ca73783-istio-podinfo\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:35.974098 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.974059    2568 generic.go:356] "Generic (PLEG): container finished" podID="6e993b0e-b03c-4eca-b18b-91356ca73783" containerID="5a6f46ca3ed33e1bf106455d77f42823a3b97c3eb6eca1b19edb05dac76a9af3" exitCode=0
Sep 11 15:06:35.974249 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.974110    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"
Sep 11 15:06:35.974249 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.974144    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m" event={"ID":"6e993b0e-b03c-4eca-b18b-91356ca73783","Type":"ContainerDied","Data":"5a6f46ca3ed33e1bf106455d77f42823a3b97c3eb6eca1b19edb05dac76a9af3"}
Sep 11 15:06:35.974249 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.974200    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m" event={"ID":"6e993b0e-b03c-4eca-b18b-91356ca73783","Type":"ContainerDied","Data":"1ee03c51d1c0e63ff80219cf10b8399dc400dd42a648324743654ab248460476"}
Sep 11 15:06:35.974249 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.974224    2568 scope.go:117] "RemoveContainer" containerID="5a6f46ca3ed33e1bf106455d77f42823a3b97c3eb6eca1b19edb05dac76a9af3"
Sep 11 15:06:35.984438 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.984419    2568 scope.go:117] "RemoveContainer" containerID="67a35fe9b1c1c931673ee8821faa0646ab573bd00f8253bda85bf5a3f49b82f2"
Sep 11 15:06:35.992606 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.992588    2568 scope.go:117] "RemoveContainer" containerID="5a6f46ca3ed33e1bf106455d77f42823a3b97c3eb6eca1b19edb05dac76a9af3"
Sep 11 15:06:35.992867 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:06:35.992846    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a6f46ca3ed33e1bf106455d77f42823a3b97c3eb6eca1b19edb05dac76a9af3\": container with ID starting with 5a6f46ca3ed33e1bf106455d77f42823a3b97c3eb6eca1b19edb05dac76a9af3 not found: ID does not exist" containerID="5a6f46ca3ed33e1bf106455d77f42823a3b97c3eb6eca1b19edb05dac76a9af3"
Sep 11 15:06:35.992941 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.992875    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a6f46ca3ed33e1bf106455d77f42823a3b97c3eb6eca1b19edb05dac76a9af3"} err="failed to get container status \"5a6f46ca3ed33e1bf106455d77f42823a3b97c3eb6eca1b19edb05dac76a9af3\": rpc error: code = NotFound desc = could not find container \"5a6f46ca3ed33e1bf106455d77f42823a3b97c3eb6eca1b19edb05dac76a9af3\": container with ID starting with 5a6f46ca3ed33e1bf106455d77f42823a3b97c3eb6eca1b19edb05dac76a9af3 not found: ID does not exist"
Sep 11 15:06:35.992941 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.992892    2568 scope.go:117] "RemoveContainer" containerID="67a35fe9b1c1c931673ee8821faa0646ab573bd00f8253bda85bf5a3f49b82f2"
Sep 11 15:06:35.993131 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:06:35.993117    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a35fe9b1c1c931673ee8821faa0646ab573bd00f8253bda85bf5a3f49b82f2\": container with ID starting with 67a35fe9b1c1c931673ee8821faa0646ab573bd00f8253bda85bf5a3f49b82f2 not found: ID does not exist" containerID="67a35fe9b1c1c931673ee8821faa0646ab573bd00f8253bda85bf5a3f49b82f2"
Sep 11 15:06:35.993169 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.993134    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a35fe9b1c1c931673ee8821faa0646ab573bd00f8253bda85bf5a3f49b82f2"} err="failed to get container status \"67a35fe9b1c1c931673ee8821faa0646ab573bd00f8253bda85bf5a3f49b82f2\": rpc error: code = NotFound desc = could not find container \"67a35fe9b1c1c931673ee8821faa0646ab573bd00f8253bda85bf5a3f49b82f2\": container with ID starting with 67a35fe9b1c1c931673ee8821faa0646ab573bd00f8253bda85bf5a3f49b82f2 not found: ID does not exist"
Sep 11 15:06:35.998485 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:35.998463    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"]
Sep 11 15:06:36.002214 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:36.002193    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-54b7f9-openshift-default-77688d4cd-hwv8m"]
Sep 11 15:06:36.662040 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:36.662010    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e993b0e-b03c-4eca-b18b-91356ca73783" path="/var/lib/kubelet/pods/6e993b0e-b03c-4eca-b18b-91356ca73783/volumes"
Sep 11 15:06:46.479605 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:46.479571    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r"]
Sep 11 15:06:46.480571 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:46.479810    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r" podUID="117a2b14-548b-464e-ad83-8f491cf41057" containerName="payload-processing" containerID="cri-o://cc2ce55c6ee48c37a9a87ae825ea3fda0e648a9773d715a5f85fca9e55e0cf6f" gracePeriod=30
Sep 11 15:06:46.486580 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:46.486547    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5"]
Sep 11 15:06:46.487039 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:46.486774    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5" podUID="9880a59b-7639-4db0-9760-1b50f3144b59" containerName="payload-pre-processing" containerID="cri-o://0de97b115799f33a024f63fb7d24ffd3476151fe5bd269334384d7fa6d6f802a" gracePeriod=30
Sep 11 15:06:46.750966 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:46.750880    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r"
Sep 11 15:06:46.777310 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:46.777279    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5"
Sep 11 15:06:46.884217 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:46.884183    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9880a59b-7639-4db0-9760-1b50f3144b59-plugins-config-volume\") pod \"9880a59b-7639-4db0-9760-1b50f3144b59\" (UID: \"9880a59b-7639-4db0-9760-1b50f3144b59\") "
Sep 11 15:06:46.884408 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:46.884269    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smsj8\" (UniqueName: \"kubernetes.io/projected/9880a59b-7639-4db0-9760-1b50f3144b59-kube-api-access-smsj8\") pod \"9880a59b-7639-4db0-9760-1b50f3144b59\" (UID: \"9880a59b-7639-4db0-9760-1b50f3144b59\") "
Sep 11 15:06:46.884408 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:46.884293    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcqgt\" (UniqueName: \"kubernetes.io/projected/117a2b14-548b-464e-ad83-8f491cf41057-kube-api-access-xcqgt\") pod \"117a2b14-548b-464e-ad83-8f491cf41057\" (UID: \"117a2b14-548b-464e-ad83-8f491cf41057\") "
Sep 11 15:06:46.884408 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:46.884317    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/117a2b14-548b-464e-ad83-8f491cf41057-plugins-config-volume\") pod \"117a2b14-548b-464e-ad83-8f491cf41057\" (UID: \"117a2b14-548b-464e-ad83-8f491cf41057\") "
Sep 11 15:06:46.884640 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:46.884609    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9880a59b-7639-4db0-9760-1b50f3144b59-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9880a59b-7639-4db0-9760-1b50f3144b59" (UID: "9880a59b-7639-4db0-9760-1b50f3144b59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:06:46.884757 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:46.884731    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117a2b14-548b-464e-ad83-8f491cf41057-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "117a2b14-548b-464e-ad83-8f491cf41057" (UID: "117a2b14-548b-464e-ad83-8f491cf41057"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:06:46.886499 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:46.886460    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9880a59b-7639-4db0-9760-1b50f3144b59-kube-api-access-smsj8" (OuterVolumeSpecName: "kube-api-access-smsj8") pod "9880a59b-7639-4db0-9760-1b50f3144b59" (UID: "9880a59b-7639-4db0-9760-1b50f3144b59"). InnerVolumeSpecName "kube-api-access-smsj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:06:46.886651 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:46.886558    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117a2b14-548b-464e-ad83-8f491cf41057-kube-api-access-xcqgt" (OuterVolumeSpecName: "kube-api-access-xcqgt") pod "117a2b14-548b-464e-ad83-8f491cf41057" (UID: "117a2b14-548b-464e-ad83-8f491cf41057"). InnerVolumeSpecName "kube-api-access-xcqgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:06:46.985758 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:46.985723    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9880a59b-7639-4db0-9760-1b50f3144b59-plugins-config-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:46.985758 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:46.985756    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smsj8\" (UniqueName: \"kubernetes.io/projected/9880a59b-7639-4db0-9760-1b50f3144b59-kube-api-access-smsj8\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:46.986012 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:46.985770    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcqgt\" (UniqueName: \"kubernetes.io/projected/117a2b14-548b-464e-ad83-8f491cf41057-kube-api-access-xcqgt\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:46.986012 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:46.985789    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/117a2b14-548b-464e-ad83-8f491cf41057-plugins-config-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:47.038669 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:47.038633    2568 generic.go:356] "Generic (PLEG): container finished" podID="117a2b14-548b-464e-ad83-8f491cf41057" containerID="cc2ce55c6ee48c37a9a87ae825ea3fda0e648a9773d715a5f85fca9e55e0cf6f" exitCode=0
Sep 11 15:06:47.038878 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:47.038677    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r"
Sep 11 15:06:47.038878 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:47.038721    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r" event={"ID":"117a2b14-548b-464e-ad83-8f491cf41057","Type":"ContainerDied","Data":"cc2ce55c6ee48c37a9a87ae825ea3fda0e648a9773d715a5f85fca9e55e0cf6f"}
Sep 11 15:06:47.038878 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:47.038774    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r" event={"ID":"117a2b14-548b-464e-ad83-8f491cf41057","Type":"ContainerDied","Data":"a4af4c48d02a99133345ed233c9647ee15e2cfd96d1d21ade5782e2538c1b142"}
Sep 11 15:06:47.038878 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:47.038803    2568 scope.go:117] "RemoveContainer" containerID="cc2ce55c6ee48c37a9a87ae825ea3fda0e648a9773d715a5f85fca9e55e0cf6f"
Sep 11 15:06:47.040236 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:47.040213    2568 generic.go:356] "Generic (PLEG): container finished" podID="9880a59b-7639-4db0-9760-1b50f3144b59" containerID="0de97b115799f33a024f63fb7d24ffd3476151fe5bd269334384d7fa6d6f802a" exitCode=0
Sep 11 15:06:47.040335 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:47.040254    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5"
Sep 11 15:06:47.040335 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:47.040303    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5" event={"ID":"9880a59b-7639-4db0-9760-1b50f3144b59","Type":"ContainerDied","Data":"0de97b115799f33a024f63fb7d24ffd3476151fe5bd269334384d7fa6d6f802a"}
Sep 11 15:06:47.040417 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:47.040360    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5" event={"ID":"9880a59b-7639-4db0-9760-1b50f3144b59","Type":"ContainerDied","Data":"2dbe401901f5819aa31f366252b7ecba3f01e61d4aa2d0fb9a21683ffb45a146"}
Sep 11 15:06:47.048409 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:47.048390    2568 scope.go:117] "RemoveContainer" containerID="cc2ce55c6ee48c37a9a87ae825ea3fda0e648a9773d715a5f85fca9e55e0cf6f"
Sep 11 15:06:47.048724 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:06:47.048657    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc2ce55c6ee48c37a9a87ae825ea3fda0e648a9773d715a5f85fca9e55e0cf6f\": container with ID starting with cc2ce55c6ee48c37a9a87ae825ea3fda0e648a9773d715a5f85fca9e55e0cf6f not found: ID does not exist" containerID="cc2ce55c6ee48c37a9a87ae825ea3fda0e648a9773d715a5f85fca9e55e0cf6f"
Sep 11 15:06:47.048724 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:47.048681    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc2ce55c6ee48c37a9a87ae825ea3fda0e648a9773d715a5f85fca9e55e0cf6f"} err="failed to get container status \"cc2ce55c6ee48c37a9a87ae825ea3fda0e648a9773d715a5f85fca9e55e0cf6f\": rpc error: code = NotFound desc = could not find container \"cc2ce55c6ee48c37a9a87ae825ea3fda0e648a9773d715a5f85fca9e55e0cf6f\": container with ID starting with cc2ce55c6ee48c37a9a87ae825ea3fda0e648a9773d715a5f85fca9e55e0cf6f not found: ID does not exist"
Sep 11 15:06:47.048724 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:47.048699    2568 scope.go:117] "RemoveContainer" containerID="0de97b115799f33a024f63fb7d24ffd3476151fe5bd269334384d7fa6d6f802a"
Sep 11 15:06:47.058099 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:47.058083    2568 scope.go:117] "RemoveContainer" containerID="0de97b115799f33a024f63fb7d24ffd3476151fe5bd269334384d7fa6d6f802a"
Sep 11 15:06:47.058357 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:06:47.058339    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0de97b115799f33a024f63fb7d24ffd3476151fe5bd269334384d7fa6d6f802a\": container with ID starting with 0de97b115799f33a024f63fb7d24ffd3476151fe5bd269334384d7fa6d6f802a not found: ID does not exist" containerID="0de97b115799f33a024f63fb7d24ffd3476151fe5bd269334384d7fa6d6f802a"
Sep 11 15:06:47.058404 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:47.058365    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0de97b115799f33a024f63fb7d24ffd3476151fe5bd269334384d7fa6d6f802a"} err="failed to get container status \"0de97b115799f33a024f63fb7d24ffd3476151fe5bd269334384d7fa6d6f802a\": rpc error: code = NotFound desc = could not find container \"0de97b115799f33a024f63fb7d24ffd3476151fe5bd269334384d7fa6d6f802a\": container with ID starting with 0de97b115799f33a024f63fb7d24ffd3476151fe5bd269334384d7fa6d6f802a not found: ID does not exist"
Sep 11 15:06:47.067059 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:47.067037    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5"]
Sep 11 15:06:47.072299 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:47.072278    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14a808-55b84f5486-24zv5"]
Sep 11 15:06:47.083245 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:47.083223    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r"]
Sep 11 15:06:47.088470 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:47.088444    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14a808-7b686594df-tvg2r"]
Sep 11 15:06:48.662792 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:48.662758    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="117a2b14-548b-464e-ad83-8f491cf41057" path="/var/lib/kubelet/pods/117a2b14-548b-464e-ad83-8f491cf41057/volumes"
Sep 11 15:06:48.663165 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:48.663131    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9880a59b-7639-4db0-9760-1b50f3144b59" path="/var/lib/kubelet/pods/9880a59b-7639-4db0-9760-1b50f3144b59/volumes"
Sep 11 15:06:54.165118 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:54.165085    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"]
Sep 11 15:06:54.165538 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:54.165288    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28" podUID="b65a3141-53e6-4feb-8628-882a06453fdd" containerName="istio-proxy" containerID="cri-o://ebe7d95d9a51eab692636a6fff405b44c98f2fed7b7357eb5f5203f9e40098e3" gracePeriod=30
Sep 11 15:06:59.414161 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.414134    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:06:59.505386 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.505309    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-istio-data\") pod \"b65a3141-53e6-4feb-8628-882a06453fdd\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") "
Sep 11 15:06:59.505386 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.505370    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-istio-envoy\") pod \"b65a3141-53e6-4feb-8628-882a06453fdd\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") "
Sep 11 15:06:59.505596 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.505395    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jk4vc\" (UniqueName: \"kubernetes.io/projected/b65a3141-53e6-4feb-8628-882a06453fdd-kube-api-access-jk4vc\") pod \"b65a3141-53e6-4feb-8628-882a06453fdd\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") "
Sep 11 15:06:59.505596 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.505423    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-workload-certs\") pod \"b65a3141-53e6-4feb-8628-882a06453fdd\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") "
Sep 11 15:06:59.505596 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.505451    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65a3141-53e6-4feb-8628-882a06453fdd-istio-token\") pod \"b65a3141-53e6-4feb-8628-882a06453fdd\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") "
Sep 11 15:06:59.505596 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.505475    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65a3141-53e6-4feb-8628-882a06453fdd-istio-podinfo\") pod \"b65a3141-53e6-4feb-8628-882a06453fdd\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") "
Sep 11 15:06:59.505596 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.505494    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-credential-socket\") pod \"b65a3141-53e6-4feb-8628-882a06453fdd\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") "
Sep 11 15:06:59.505596 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.505550    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65a3141-53e6-4feb-8628-882a06453fdd-istiod-ca-cert\") pod \"b65a3141-53e6-4feb-8628-882a06453fdd\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") "
Sep 11 15:06:59.505596 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.505581    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-workload-socket\") pod \"b65a3141-53e6-4feb-8628-882a06453fdd\" (UID: \"b65a3141-53e6-4feb-8628-882a06453fdd\") "
Sep 11 15:06:59.506007 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.505621    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-istio-data" (OuterVolumeSpecName: "istio-data") pod "b65a3141-53e6-4feb-8628-882a06453fdd" (UID: "b65a3141-53e6-4feb-8628-882a06453fdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:06:59.506007 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.505683    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b65a3141-53e6-4feb-8628-882a06453fdd" (UID: "b65a3141-53e6-4feb-8628-882a06453fdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:06:59.506007 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.505912    2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-istio-data\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:59.506007 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.505931    2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-workload-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:59.506007 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.505913    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b65a3141-53e6-4feb-8628-882a06453fdd" (UID: "b65a3141-53e6-4feb-8628-882a06453fdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:06:59.506007 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.505956    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b65a3141-53e6-4feb-8628-882a06453fdd" (UID: "b65a3141-53e6-4feb-8628-882a06453fdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:06:59.506007 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.506005    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b65a3141-53e6-4feb-8628-882a06453fdd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b65a3141-53e6-4feb-8628-882a06453fdd" (UID: "b65a3141-53e6-4feb-8628-882a06453fdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:06:59.507708 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.507684    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65a3141-53e6-4feb-8628-882a06453fdd-kube-api-access-jk4vc" (OuterVolumeSpecName: "kube-api-access-jk4vc") pod "b65a3141-53e6-4feb-8628-882a06453fdd" (UID: "b65a3141-53e6-4feb-8628-882a06453fdd"). InnerVolumeSpecName "kube-api-access-jk4vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:06:59.507838 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.507732    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65a3141-53e6-4feb-8628-882a06453fdd-istio-token" (OuterVolumeSpecName: "istio-token") pod "b65a3141-53e6-4feb-8628-882a06453fdd" (UID: "b65a3141-53e6-4feb-8628-882a06453fdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:06:59.507838 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.507746    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b65a3141-53e6-4feb-8628-882a06453fdd" (UID: "b65a3141-53e6-4feb-8628-882a06453fdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:06:59.508121 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.508103    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b65a3141-53e6-4feb-8628-882a06453fdd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b65a3141-53e6-4feb-8628-882a06453fdd" (UID: "b65a3141-53e6-4feb-8628-882a06453fdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 15:06:59.606713 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.606683    2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65a3141-53e6-4feb-8628-882a06453fdd-istio-token\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:59.606713 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.606707    2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65a3141-53e6-4feb-8628-882a06453fdd-istio-podinfo\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:59.606713 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.606718    2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-credential-socket\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:59.606925 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.606728    2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65a3141-53e6-4feb-8628-882a06453fdd-istiod-ca-cert\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:59.606925 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.606737    2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-workload-socket\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:59.606925 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.606745    2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65a3141-53e6-4feb-8628-882a06453fdd-istio-envoy\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:06:59.606925 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:06:59.606754    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk4vc\" (UniqueName: \"kubernetes.io/projected/b65a3141-53e6-4feb-8628-882a06453fdd-kube-api-access-jk4vc\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:00.124326 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:00.124290    2568 generic.go:356] "Generic (PLEG): container finished" podID="b65a3141-53e6-4feb-8628-882a06453fdd" containerID="ebe7d95d9a51eab692636a6fff405b44c98f2fed7b7357eb5f5203f9e40098e3" exitCode=0
Sep 11 15:07:00.124493 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:00.124344    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28" event={"ID":"b65a3141-53e6-4feb-8628-882a06453fdd","Type":"ContainerDied","Data":"ebe7d95d9a51eab692636a6fff405b44c98f2fed7b7357eb5f5203f9e40098e3"}
Sep 11 15:07:00.124493 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:00.124362    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"
Sep 11 15:07:00.124493 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:00.124405    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28" event={"ID":"b65a3141-53e6-4feb-8628-882a06453fdd","Type":"ContainerDied","Data":"fa199199c1f0d52b835fa4c9db8d1768f05c326e87c208e144132403e630be2f"}
Sep 11 15:07:00.124493 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:00.124426    2568 scope.go:117] "RemoveContainer" containerID="ebe7d95d9a51eab692636a6fff405b44c98f2fed7b7357eb5f5203f9e40098e3"
Sep 11 15:07:00.133984 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:00.133966    2568 scope.go:117] "RemoveContainer" containerID="7998205db540b48aa8fe1252653fa3ad6fb9c15fdff552e15d20a36107a982fe"
Sep 11 15:07:00.141896 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:00.141879    2568 scope.go:117] "RemoveContainer" containerID="ebe7d95d9a51eab692636a6fff405b44c98f2fed7b7357eb5f5203f9e40098e3"
Sep 11 15:07:00.142161 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:07:00.142142    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe7d95d9a51eab692636a6fff405b44c98f2fed7b7357eb5f5203f9e40098e3\": container with ID starting with ebe7d95d9a51eab692636a6fff405b44c98f2fed7b7357eb5f5203f9e40098e3 not found: ID does not exist" containerID="ebe7d95d9a51eab692636a6fff405b44c98f2fed7b7357eb5f5203f9e40098e3"
Sep 11 15:07:00.142238 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:00.142173    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe7d95d9a51eab692636a6fff405b44c98f2fed7b7357eb5f5203f9e40098e3"} err="failed to get container status \"ebe7d95d9a51eab692636a6fff405b44c98f2fed7b7357eb5f5203f9e40098e3\": rpc error: code = NotFound desc = could not find container \"ebe7d95d9a51eab692636a6fff405b44c98f2fed7b7357eb5f5203f9e40098e3\": container with ID starting with ebe7d95d9a51eab692636a6fff405b44c98f2fed7b7357eb5f5203f9e40098e3 not found: ID does not exist"
Sep 11 15:07:00.142238 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:00.142195    2568 scope.go:117] "RemoveContainer" containerID="7998205db540b48aa8fe1252653fa3ad6fb9c15fdff552e15d20a36107a982fe"
Sep 11 15:07:00.142440 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:07:00.142425    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7998205db540b48aa8fe1252653fa3ad6fb9c15fdff552e15d20a36107a982fe\": container with ID starting with 7998205db540b48aa8fe1252653fa3ad6fb9c15fdff552e15d20a36107a982fe not found: ID does not exist" containerID="7998205db540b48aa8fe1252653fa3ad6fb9c15fdff552e15d20a36107a982fe"
Sep 11 15:07:00.142476 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:00.142446    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7998205db540b48aa8fe1252653fa3ad6fb9c15fdff552e15d20a36107a982fe"} err="failed to get container status \"7998205db540b48aa8fe1252653fa3ad6fb9c15fdff552e15d20a36107a982fe\": rpc error: code = NotFound desc = could not find container \"7998205db540b48aa8fe1252653fa3ad6fb9c15fdff552e15d20a36107a982fe\": container with ID starting with 7998205db540b48aa8fe1252653fa3ad6fb9c15fdff552e15d20a36107a982fe not found: ID does not exist"
Sep 11 15:07:00.146761 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:00.146742    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"]
Sep 11 15:07:00.150811 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:00.150790    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-14a808-openshift-default-767dc55cdb-52h28"]
Sep 11 15:07:00.661197 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:00.661164    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b65a3141-53e6-4feb-8628-882a06453fdd" path="/var/lib/kubelet/pods/b65a3141-53e6-4feb-8628-882a06453fdd/volumes"
Sep 11 15:07:05.189995 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:05.189954    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr"]
Sep 11 15:07:05.190428 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:05.190169    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr" podUID="2dbcdde6-9675-486d-b368-2aef2b6bed05" containerName="payload-pre-processing" containerID="cri-o://494506fba4ebbf30aa46c83015613ca2bdaa7002616edf16859a90384de5eab5" gracePeriod=30
Sep 11 15:07:05.449359 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:05.449296    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr"
Sep 11 15:07:05.560697 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:05.560652    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbcdde6-9675-486d-b368-2aef2b6bed05-plugins-config-volume\") pod \"2dbcdde6-9675-486d-b368-2aef2b6bed05\" (UID: \"2dbcdde6-9675-486d-b368-2aef2b6bed05\") "
Sep 11 15:07:05.560923 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:05.560735    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7md7\" (UniqueName: \"kubernetes.io/projected/2dbcdde6-9675-486d-b368-2aef2b6bed05-kube-api-access-v7md7\") pod \"2dbcdde6-9675-486d-b368-2aef2b6bed05\" (UID: \"2dbcdde6-9675-486d-b368-2aef2b6bed05\") "
Sep 11 15:07:05.561059 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:05.561036    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dbcdde6-9675-486d-b368-2aef2b6bed05-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2dbcdde6-9675-486d-b368-2aef2b6bed05" (UID: "2dbcdde6-9675-486d-b368-2aef2b6bed05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:07:05.562825 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:05.562794    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dbcdde6-9675-486d-b368-2aef2b6bed05-kube-api-access-v7md7" (OuterVolumeSpecName: "kube-api-access-v7md7") pod "2dbcdde6-9675-486d-b368-2aef2b6bed05" (UID: "2dbcdde6-9675-486d-b368-2aef2b6bed05"). InnerVolumeSpecName "kube-api-access-v7md7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:07:05.661429 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:05.661392    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbcdde6-9675-486d-b368-2aef2b6bed05-plugins-config-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:05.661429 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:05.661423    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7md7\" (UniqueName: \"kubernetes.io/projected/2dbcdde6-9675-486d-b368-2aef2b6bed05-kube-api-access-v7md7\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:06.156984 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:06.156944    2568 generic.go:356] "Generic (PLEG): container finished" podID="2dbcdde6-9675-486d-b368-2aef2b6bed05" containerID="494506fba4ebbf30aa46c83015613ca2bdaa7002616edf16859a90384de5eab5" exitCode=0
Sep 11 15:07:06.157165 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:06.156990    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr"
Sep 11 15:07:06.157165 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:06.157031    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr" event={"ID":"2dbcdde6-9675-486d-b368-2aef2b6bed05","Type":"ContainerDied","Data":"494506fba4ebbf30aa46c83015613ca2bdaa7002616edf16859a90384de5eab5"}
Sep 11 15:07:06.157165 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:06.157083    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr" event={"ID":"2dbcdde6-9675-486d-b368-2aef2b6bed05","Type":"ContainerDied","Data":"39141a37ceb35de82a2b7feb25d5c81aa67337fa8c655f536c30584a2d02a52d"}
Sep 11 15:07:06.157165 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:06.157105    2568 scope.go:117] "RemoveContainer" containerID="494506fba4ebbf30aa46c83015613ca2bdaa7002616edf16859a90384de5eab5"
Sep 11 15:07:06.166525 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:06.166508    2568 scope.go:117] "RemoveContainer" containerID="494506fba4ebbf30aa46c83015613ca2bdaa7002616edf16859a90384de5eab5"
Sep 11 15:07:06.166777 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:07:06.166757    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494506fba4ebbf30aa46c83015613ca2bdaa7002616edf16859a90384de5eab5\": container with ID starting with 494506fba4ebbf30aa46c83015613ca2bdaa7002616edf16859a90384de5eab5 not found: ID does not exist" containerID="494506fba4ebbf30aa46c83015613ca2bdaa7002616edf16859a90384de5eab5"
Sep 11 15:07:06.166836 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:06.166787    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494506fba4ebbf30aa46c83015613ca2bdaa7002616edf16859a90384de5eab5"} err="failed to get container status \"494506fba4ebbf30aa46c83015613ca2bdaa7002616edf16859a90384de5eab5\": rpc error: code = NotFound desc = could not find container \"494506fba4ebbf30aa46c83015613ca2bdaa7002616edf16859a90384de5eab5\": container with ID starting with 494506fba4ebbf30aa46c83015613ca2bdaa7002616edf16859a90384de5eab5 not found: ID does not exist"
Sep 11 15:07:06.182092 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:06.182068    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr"]
Sep 11 15:07:06.185588 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:06.185568    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4e39b-6c76f7f8-9t6nr"]
Sep 11 15:07:06.662786 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:06.662748    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dbcdde6-9675-486d-b368-2aef2b6bed05" path="/var/lib/kubelet/pods/2dbcdde6-9675-486d-b368-2aef2b6bed05/volumes"
Sep 11 15:07:12.320158 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:12.320117    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"]
Sep 11 15:07:12.320604 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:12.320381    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph" podUID="51e90696-abd3-4ccd-b528-df0eea3b2023" containerName="istio-proxy" containerID="cri-o://c37c4333dc76e3c7f88dedf8a449e302b8a5dd5502d0959b5b00884cbefa8db6" gracePeriod=30
Sep 11 15:07:17.213757 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.213671    2568 patch_prober.go:28] interesting pod/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 15:07:17.213757 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.213744    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph" podUID="51e90696-abd3-4ccd-b528-df0eea3b2023" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 15:07:17.596178 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.596154    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 15:07:17.674203 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.674162    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-podinfo\") pod \"51e90696-abd3-4ccd-b528-df0eea3b2023\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") "
Sep 11 15:07:17.674203 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.674208    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsq57\" (UniqueName: \"kubernetes.io/projected/51e90696-abd3-4ccd-b528-df0eea3b2023-kube-api-access-tsq57\") pod \"51e90696-abd3-4ccd-b528-df0eea3b2023\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") "
Sep 11 15:07:17.674409 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.674325    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-token\") pod \"51e90696-abd3-4ccd-b528-df0eea3b2023\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") "
Sep 11 15:07:17.674409 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.674358    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-workload-certs\") pod \"51e90696-abd3-4ccd-b528-df0eea3b2023\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") "
Sep 11 15:07:17.674409 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.674378    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-data\") pod \"51e90696-abd3-4ccd-b528-df0eea3b2023\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") "
Sep 11 15:07:17.674575 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.674423    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-credential-socket\") pod \"51e90696-abd3-4ccd-b528-df0eea3b2023\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") "
Sep 11 15:07:17.674575 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.674456    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-envoy\") pod \"51e90696-abd3-4ccd-b528-df0eea3b2023\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") "
Sep 11 15:07:17.674575 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.674480    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e90696-abd3-4ccd-b528-df0eea3b2023-istiod-ca-cert\") pod \"51e90696-abd3-4ccd-b528-df0eea3b2023\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") "
Sep 11 15:07:17.674575 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.674556    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-workload-socket\") pod \"51e90696-abd3-4ccd-b528-df0eea3b2023\" (UID: \"51e90696-abd3-4ccd-b528-df0eea3b2023\") "
Sep 11 15:07:17.674784 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.674615    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "51e90696-abd3-4ccd-b528-df0eea3b2023" (UID: "51e90696-abd3-4ccd-b528-df0eea3b2023"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:07:17.674784 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.674645    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-data" (OuterVolumeSpecName: "istio-data") pod "51e90696-abd3-4ccd-b528-df0eea3b2023" (UID: "51e90696-abd3-4ccd-b528-df0eea3b2023"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:07:17.674934 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.674871    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "51e90696-abd3-4ccd-b528-df0eea3b2023" (UID: "51e90696-abd3-4ccd-b528-df0eea3b2023"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:07:17.674934 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.674909    2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-workload-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:17.674934 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.674931    2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-data\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:17.675089 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.674928    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e90696-abd3-4ccd-b528-df0eea3b2023-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "51e90696-abd3-4ccd-b528-df0eea3b2023" (UID: "51e90696-abd3-4ccd-b528-df0eea3b2023"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:07:17.675089 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.675025    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "51e90696-abd3-4ccd-b528-df0eea3b2023" (UID: "51e90696-abd3-4ccd-b528-df0eea3b2023"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:07:17.676534 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.676503    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "51e90696-abd3-4ccd-b528-df0eea3b2023" (UID: "51e90696-abd3-4ccd-b528-df0eea3b2023"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:07:17.676637 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.676530    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e90696-abd3-4ccd-b528-df0eea3b2023-kube-api-access-tsq57" (OuterVolumeSpecName: "kube-api-access-tsq57") pod "51e90696-abd3-4ccd-b528-df0eea3b2023" (UID: "51e90696-abd3-4ccd-b528-df0eea3b2023"). InnerVolumeSpecName "kube-api-access-tsq57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:07:17.676777 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.676754    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-token" (OuterVolumeSpecName: "istio-token") pod "51e90696-abd3-4ccd-b528-df0eea3b2023" (UID: "51e90696-abd3-4ccd-b528-df0eea3b2023"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:07:17.676847 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.676754    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "51e90696-abd3-4ccd-b528-df0eea3b2023" (UID: "51e90696-abd3-4ccd-b528-df0eea3b2023"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 15:07:17.776372 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.776348    2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-workload-socket\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:17.776372 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.776371    2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-podinfo\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:17.776514 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.776381    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsq57\" (UniqueName: \"kubernetes.io/projected/51e90696-abd3-4ccd-b528-df0eea3b2023-kube-api-access-tsq57\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:17.776514 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.776390    2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-token\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:17.776514 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.776398    2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-credential-socket\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:17.776514 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.776406    2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e90696-abd3-4ccd-b528-df0eea3b2023-istio-envoy\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:17.776514 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:17.776415    2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e90696-abd3-4ccd-b528-df0eea3b2023-istiod-ca-cert\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:18.221867 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:18.221751    2568 generic.go:356] "Generic (PLEG): container finished" podID="51e90696-abd3-4ccd-b528-df0eea3b2023" containerID="c37c4333dc76e3c7f88dedf8a449e302b8a5dd5502d0959b5b00884cbefa8db6" exitCode=0
Sep 11 15:07:18.221867 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:18.221807    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"
Sep 11 15:07:18.221867 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:18.221857    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph" event={"ID":"51e90696-abd3-4ccd-b528-df0eea3b2023","Type":"ContainerDied","Data":"c37c4333dc76e3c7f88dedf8a449e302b8a5dd5502d0959b5b00884cbefa8db6"}
Sep 11 15:07:18.222458 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:18.221919    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph" event={"ID":"51e90696-abd3-4ccd-b528-df0eea3b2023","Type":"ContainerDied","Data":"a2080fc07de3ec06d186aab28d3ce77e830c85899964411242567d40699fefd8"}
Sep 11 15:07:18.222458 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:18.221943    2568 scope.go:117] "RemoveContainer" containerID="c37c4333dc76e3c7f88dedf8a449e302b8a5dd5502d0959b5b00884cbefa8db6"
Sep 11 15:07:18.231493 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:18.231471    2568 scope.go:117] "RemoveContainer" containerID="bdc9afc25096dcdb57f73995eea5a007211ddac2d837e53c477bb4671bc40491"
Sep 11 15:07:18.239670 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:18.239654    2568 scope.go:117] "RemoveContainer" containerID="c37c4333dc76e3c7f88dedf8a449e302b8a5dd5502d0959b5b00884cbefa8db6"
Sep 11 15:07:18.239948 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:07:18.239929    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37c4333dc76e3c7f88dedf8a449e302b8a5dd5502d0959b5b00884cbefa8db6\": container with ID starting with c37c4333dc76e3c7f88dedf8a449e302b8a5dd5502d0959b5b00884cbefa8db6 not found: ID does not exist" containerID="c37c4333dc76e3c7f88dedf8a449e302b8a5dd5502d0959b5b00884cbefa8db6"
Sep 11 15:07:18.240015 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:18.239955    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37c4333dc76e3c7f88dedf8a449e302b8a5dd5502d0959b5b00884cbefa8db6"} err="failed to get container status \"c37c4333dc76e3c7f88dedf8a449e302b8a5dd5502d0959b5b00884cbefa8db6\": rpc error: code = NotFound desc = could not find container \"c37c4333dc76e3c7f88dedf8a449e302b8a5dd5502d0959b5b00884cbefa8db6\": container with ID starting with c37c4333dc76e3c7f88dedf8a449e302b8a5dd5502d0959b5b00884cbefa8db6 not found: ID does not exist"
Sep 11 15:07:18.240015 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:18.239973    2568 scope.go:117] "RemoveContainer" containerID="bdc9afc25096dcdb57f73995eea5a007211ddac2d837e53c477bb4671bc40491"
Sep 11 15:07:18.240218 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:07:18.240199    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc9afc25096dcdb57f73995eea5a007211ddac2d837e53c477bb4671bc40491\": container with ID starting with bdc9afc25096dcdb57f73995eea5a007211ddac2d837e53c477bb4671bc40491 not found: ID does not exist" containerID="bdc9afc25096dcdb57f73995eea5a007211ddac2d837e53c477bb4671bc40491"
Sep 11 15:07:18.240260 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:18.240223    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc9afc25096dcdb57f73995eea5a007211ddac2d837e53c477bb4671bc40491"} err="failed to get container status \"bdc9afc25096dcdb57f73995eea5a007211ddac2d837e53c477bb4671bc40491\": rpc error: code = NotFound desc = could not find container \"bdc9afc25096dcdb57f73995eea5a007211ddac2d837e53c477bb4671bc40491\": container with ID starting with bdc9afc25096dcdb57f73995eea5a007211ddac2d837e53c477bb4671bc40491 not found: ID does not exist"
Sep 11 15:07:18.244734 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:18.244712    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"]
Sep 11 15:07:18.248572 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:18.248551    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c4e39b-openshift-default-65d76db66b-fxxph"]
Sep 11 15:07:18.664144 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:18.664110    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51e90696-abd3-4ccd-b528-df0eea3b2023" path="/var/lib/kubelet/pods/51e90696-abd3-4ccd-b528-df0eea3b2023/volumes"
Sep 11 15:07:23.300021 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:23.299978    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"]
Sep 11 15:07:23.300443 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:23.300234    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv" podUID="0abfb50a-d22e-4dc4-b8af-814179a37f9c" containerName="maas-api" containerID="cri-o://f6e3b24c1a8f30d0b5ce1c40f2fbef01e39ca4312e8eab2b2601c54380148a1c" gracePeriod=30
Sep 11 15:07:23.548198 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:23.548165    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"
Sep 11 15:07:23.627567 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:23.627482    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0abfb50a-d22e-4dc4-b8af-814179a37f9c-openshift-service-ca\") pod \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\" (UID: \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\") "
Sep 11 15:07:23.627567 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:23.627549    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0abfb50a-d22e-4dc4-b8af-814179a37f9c-metrics-certs\") pod \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\" (UID: \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\") "
Sep 11 15:07:23.627802 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:23.627599    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0abfb50a-d22e-4dc4-b8af-814179a37f9c-maas-api-tls\") pod \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\" (UID: \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\") "
Sep 11 15:07:23.627802 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:23.627615    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlbss\" (UniqueName: \"kubernetes.io/projected/0abfb50a-d22e-4dc4-b8af-814179a37f9c-kube-api-access-mlbss\") pod \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\" (UID: \"0abfb50a-d22e-4dc4-b8af-814179a37f9c\") "
Sep 11 15:07:23.627958 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:23.627924    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0abfb50a-d22e-4dc4-b8af-814179a37f9c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0abfb50a-d22e-4dc4-b8af-814179a37f9c" (UID: "0abfb50a-d22e-4dc4-b8af-814179a37f9c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:07:23.629763 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:23.629731    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0abfb50a-d22e-4dc4-b8af-814179a37f9c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0abfb50a-d22e-4dc4-b8af-814179a37f9c" (UID: "0abfb50a-d22e-4dc4-b8af-814179a37f9c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:07:23.629922 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:23.629857    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0abfb50a-d22e-4dc4-b8af-814179a37f9c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0abfb50a-d22e-4dc4-b8af-814179a37f9c" (UID: "0abfb50a-d22e-4dc4-b8af-814179a37f9c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:07:23.629922 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:23.629896    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0abfb50a-d22e-4dc4-b8af-814179a37f9c-kube-api-access-mlbss" (OuterVolumeSpecName: "kube-api-access-mlbss") pod "0abfb50a-d22e-4dc4-b8af-814179a37f9c" (UID: "0abfb50a-d22e-4dc4-b8af-814179a37f9c"). InnerVolumeSpecName "kube-api-access-mlbss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:07:23.728646 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:23.728607    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0abfb50a-d22e-4dc4-b8af-814179a37f9c-metrics-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:23.728646 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:23.728640    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0abfb50a-d22e-4dc4-b8af-814179a37f9c-maas-api-tls\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:23.728646 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:23.728651    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlbss\" (UniqueName: \"kubernetes.io/projected/0abfb50a-d22e-4dc4-b8af-814179a37f9c-kube-api-access-mlbss\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:23.728917 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:23.728664    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0abfb50a-d22e-4dc4-b8af-814179a37f9c-openshift-service-ca\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:24.253887 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:24.253847    2568 generic.go:356] "Generic (PLEG): container finished" podID="0abfb50a-d22e-4dc4-b8af-814179a37f9c" containerID="f6e3b24c1a8f30d0b5ce1c40f2fbef01e39ca4312e8eab2b2601c54380148a1c" exitCode=0
Sep 11 15:07:24.254083 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:24.253911    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"
Sep 11 15:07:24.254083 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:24.253932    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv" event={"ID":"0abfb50a-d22e-4dc4-b8af-814179a37f9c","Type":"ContainerDied","Data":"f6e3b24c1a8f30d0b5ce1c40f2fbef01e39ca4312e8eab2b2601c54380148a1c"}
Sep 11 15:07:24.254083 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:24.253988    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv" event={"ID":"0abfb50a-d22e-4dc4-b8af-814179a37f9c","Type":"ContainerDied","Data":"77ed6ad4c7384b63035e67e42a579efee04b7518673265a77a5a39c4ece77df2"}
Sep 11 15:07:24.254083 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:24.254010    2568 scope.go:117] "RemoveContainer" containerID="f6e3b24c1a8f30d0b5ce1c40f2fbef01e39ca4312e8eab2b2601c54380148a1c"
Sep 11 15:07:24.263257 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:24.263240    2568 scope.go:117] "RemoveContainer" containerID="f6e3b24c1a8f30d0b5ce1c40f2fbef01e39ca4312e8eab2b2601c54380148a1c"
Sep 11 15:07:24.263521 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:07:24.263502    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e3b24c1a8f30d0b5ce1c40f2fbef01e39ca4312e8eab2b2601c54380148a1c\": container with ID starting with f6e3b24c1a8f30d0b5ce1c40f2fbef01e39ca4312e8eab2b2601c54380148a1c not found: ID does not exist" containerID="f6e3b24c1a8f30d0b5ce1c40f2fbef01e39ca4312e8eab2b2601c54380148a1c"
Sep 11 15:07:24.263596 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:24.263526    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e3b24c1a8f30d0b5ce1c40f2fbef01e39ca4312e8eab2b2601c54380148a1c"} err="failed to get container status \"f6e3b24c1a8f30d0b5ce1c40f2fbef01e39ca4312e8eab2b2601c54380148a1c\": rpc error: code = NotFound desc = could not find container \"f6e3b24c1a8f30d0b5ce1c40f2fbef01e39ca4312e8eab2b2601c54380148a1c\": container with ID starting with f6e3b24c1a8f30d0b5ce1c40f2fbef01e39ca4312e8eab2b2601c54380148a1c not found: ID does not exist"
Sep 11 15:07:24.276124 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:24.276096    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"]
Sep 11 15:07:24.279828 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:24.279796    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0e28e-84bc7c4d66-r5grv"]
Sep 11 15:07:24.662402 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:24.662369    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0abfb50a-d22e-4dc4-b8af-814179a37f9c" path="/var/lib/kubelet/pods/0abfb50a-d22e-4dc4-b8af-814179a37f9c/volumes"
Sep 11 15:07:25.096786 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:25.096752    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"]
Sep 11 15:07:34.963692 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.963660    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"]
Sep 11 15:07:34.964179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964053    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51e90696-abd3-4ccd-b528-df0eea3b2023" containerName="istio-proxy"
Sep 11 15:07:34.964179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964066    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="51e90696-abd3-4ccd-b528-df0eea3b2023" containerName="istio-proxy"
Sep 11 15:07:34.964179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964078    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7264546b-16e5-4b54-9542-5943919be700" containerName="payload-pre-processing"
Sep 11 15:07:34.964179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964084    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7264546b-16e5-4b54-9542-5943919be700" containerName="payload-pre-processing"
Sep 11 15:07:34.964179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964095    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="117a2b14-548b-464e-ad83-8f491cf41057" containerName="payload-processing"
Sep 11 15:07:34.964179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964101    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="117a2b14-548b-464e-ad83-8f491cf41057" containerName="payload-processing"
Sep 11 15:07:34.964179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964113    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b65a3141-53e6-4feb-8628-882a06453fdd" containerName="istio-proxy"
Sep 11 15:07:34.964179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964118    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b65a3141-53e6-4feb-8628-882a06453fdd" containerName="istio-proxy"
Sep 11 15:07:34.964179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964127    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74e3498c-b895-4e07-82af-9586e0ef670d" containerName="payload-pre-processing"
Sep 11 15:07:34.964179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964132    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="74e3498c-b895-4e07-82af-9586e0ef670d" containerName="payload-pre-processing"
Sep 11 15:07:34.964179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964138    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e993b0e-b03c-4eca-b18b-91356ca73783" containerName="istio-proxy"
Sep 11 15:07:34.964179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964143    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e993b0e-b03c-4eca-b18b-91356ca73783" containerName="istio-proxy"
Sep 11 15:07:34.964179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964149    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0abfb50a-d22e-4dc4-b8af-814179a37f9c" containerName="maas-api"
Sep 11 15:07:34.964179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964154    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0abfb50a-d22e-4dc4-b8af-814179a37f9c" containerName="maas-api"
Sep 11 15:07:34.964179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964162    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b65a3141-53e6-4feb-8628-882a06453fdd" containerName="istio-proxy"
Sep 11 15:07:34.964179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964167    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b65a3141-53e6-4feb-8628-882a06453fdd" containerName="istio-proxy"
Sep 11 15:07:34.964179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964173    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dbcdde6-9675-486d-b368-2aef2b6bed05" containerName="payload-pre-processing"
Sep 11 15:07:34.964179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964178    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dbcdde6-9675-486d-b368-2aef2b6bed05" containerName="payload-pre-processing"
Sep 11 15:07:34.964179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964187    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e993b0e-b03c-4eca-b18b-91356ca73783" containerName="istio-proxy"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964193    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e993b0e-b03c-4eca-b18b-91356ca73783" containerName="istio-proxy"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964201    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dead80a8-6b34-4408-ac1b-867c2ec0b91f" containerName="payload-processing"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964206    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dead80a8-6b34-4408-ac1b-867c2ec0b91f" containerName="payload-processing"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964212    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9880a59b-7639-4db0-9760-1b50f3144b59" containerName="payload-pre-processing"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964217    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9880a59b-7639-4db0-9760-1b50f3144b59" containerName="payload-pre-processing"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964225    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd30c533-7634-4d21-a6e0-249dbba1e43c" containerName="maas-api"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964231    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd30c533-7634-4d21-a6e0-249dbba1e43c" containerName="maas-api"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964290    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7264546b-16e5-4b54-9542-5943919be700" containerName="payload-pre-processing"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964298    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dbcdde6-9675-486d-b368-2aef2b6bed05" containerName="payload-pre-processing"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964303    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e993b0e-b03c-4eca-b18b-91356ca73783" containerName="istio-proxy"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964309    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b65a3141-53e6-4feb-8628-882a06453fdd" containerName="istio-proxy"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964314    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e993b0e-b03c-4eca-b18b-91356ca73783" containerName="istio-proxy"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964319    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b65a3141-53e6-4feb-8628-882a06453fdd" containerName="istio-proxy"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964327    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dead80a8-6b34-4408-ac1b-867c2ec0b91f" containerName="payload-processing"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964333    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="51e90696-abd3-4ccd-b528-df0eea3b2023" containerName="istio-proxy"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964338    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd30c533-7634-4d21-a6e0-249dbba1e43c" containerName="maas-api"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964343    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0abfb50a-d22e-4dc4-b8af-814179a37f9c" containerName="maas-api"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964348    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="117a2b14-548b-464e-ad83-8f491cf41057" containerName="payload-processing"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964353    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="74e3498c-b895-4e07-82af-9586e0ef670d" containerName="payload-pre-processing"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964358    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="51e90696-abd3-4ccd-b528-df0eea3b2023" containerName="istio-proxy"
Sep 11 15:07:34.964794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.964364    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9880a59b-7639-4db0-9760-1b50f3144b59" containerName="payload-pre-processing"
Sep 11 15:07:34.968919 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.968891    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:34.971807 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.971787    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-48898f\"/\"sub-test-model-48898f-kserve-self-signed-certs\""
Sep 11 15:07:34.976742 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:34.976722    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"]
Sep 11 15:07:35.134513 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:35.134477    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t87hj\" (UniqueName: \"kubernetes.io/projected/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-kube-api-access-t87hj\") pod \"sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") " pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:35.134513 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:35.134516    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-model-cache\") pod \"sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") " pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:35.134721 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:35.134604    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-dshm\") pod \"sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") " pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:35.134721 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:35.134658    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-tmp-dir\") pod \"sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") " pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:35.134721 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:35.134702    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-home\") pod \"sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") " pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:35.134859 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:35.134724    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-tls-certs\") pod \"sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") " pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:35.235919 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:35.235802    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t87hj\" (UniqueName: \"kubernetes.io/projected/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-kube-api-access-t87hj\") pod \"sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") " pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:35.236084 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:35.235947    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-model-cache\") pod \"sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") " pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:35.236084 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:35.235996    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-dshm\") pod \"sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") " pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:35.236084 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:35.236027    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-tmp-dir\") pod \"sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") " pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:35.236084 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:35.236044    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-home\") pod \"sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") " pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:35.236084 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:35.236060    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-tls-certs\") pod \"sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") " pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:35.236433 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:35.236408    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-model-cache\") pod \"sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") " pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:35.236502 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:35.236419    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-tmp-dir\") pod \"sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") " pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:35.236544 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:35.236504    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-home\") pod \"sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") " pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:35.238323 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:35.238301    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-dshm\") pod \"sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") " pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:35.238506 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:35.238490    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-tls-certs\") pod \"sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") " pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:35.243928 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:35.243904    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t87hj\" (UniqueName: \"kubernetes.io/projected/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-kube-api-access-t87hj\") pod \"sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") " pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:35.278808 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:35.278769    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:35.554867 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:35.554842    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"]
Sep 11 15:07:35.557140 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:07:35.557113    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a6ab11c_425d_4d1a_9a7b_b18b997bde11.slice/crio-3a88bb88690cb8fd789ca12637c7110f68e51c4c146eed9e69e6775e30e2253b WatchSource:0}: Error finding container 3a88bb88690cb8fd789ca12637c7110f68e51c4c146eed9e69e6775e30e2253b: Status 404 returned error can't find the container with id 3a88bb88690cb8fd789ca12637c7110f68e51c4c146eed9e69e6775e30e2253b
Sep 11 15:07:36.313337 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:36.313298    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9" event={"ID":"3a6ab11c-425d-4d1a-9a7b-b18b997bde11","Type":"ContainerStarted","Data":"741e917bb615874bbf0360c41b13b2ebf79930f4553457705a4b0fdf84f9e22c"}
Sep 11 15:07:36.313859 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:36.313349    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9" event={"ID":"3a6ab11c-425d-4d1a-9a7b-b18b997bde11","Type":"ContainerStarted","Data":"3a88bb88690cb8fd789ca12637c7110f68e51c4c146eed9e69e6775e30e2253b"}
Sep 11 15:07:36.333988 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:36.333938    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9" podStartSLOduration=2.333923915 podStartE2EDuration="2.333923915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:07:36.331905799 +0000 UTC m=+1636.120170098" watchObservedRunningTime="2026-09-11 15:07:36.333923915 +0000 UTC m=+1636.122188211"
Sep 11 15:07:40.278371 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:40.278321    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5" podUID="0903a7b6-7261-49d0-b3fc-f839f9f8ecdf" containerName="main" containerID="cri-o://39b7eaa1504230d58c0ecb519c66a56ececde15e37609e4ad013a86d459b66ff" gracePeriod=45
Sep 11 15:07:40.532713 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:40.532658    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:07:40.586433 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:40.586405    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-tmp-dir\") pod \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") "
Sep 11 15:07:40.586585 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:40.586465    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-tls-certs\") pod \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") "
Sep 11 15:07:40.586585 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:40.586497    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-dshm\") pod \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") "
Sep 11 15:07:40.586585 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:40.586528    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-model-cache\") pod \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") "
Sep 11 15:07:40.586585 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:40.586547    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-home\") pod \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") "
Sep 11 15:07:40.586585 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:40.586581    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdvn6\" (UniqueName: \"kubernetes.io/projected/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-kube-api-access-kdvn6\") pod \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\" (UID: \"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf\") "
Sep 11 15:07:40.586896 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:40.586659    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0903a7b6-7261-49d0-b3fc-f839f9f8ecdf" (UID: "0903a7b6-7261-49d0-b3fc-f839f9f8ecdf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:07:40.586896 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:40.586844    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-home" (OuterVolumeSpecName: "home") pod "0903a7b6-7261-49d0-b3fc-f839f9f8ecdf" (UID: "0903a7b6-7261-49d0-b3fc-f839f9f8ecdf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:07:40.586896 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:40.586858    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-model-cache" (OuterVolumeSpecName: "model-cache") pod "0903a7b6-7261-49d0-b3fc-f839f9f8ecdf" (UID: "0903a7b6-7261-49d0-b3fc-f839f9f8ecdf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:07:40.587066 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:40.586977    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-model-cache\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:40.587066 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:40.586993    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-home\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:40.587066 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:40.587005    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-tmp-dir\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:40.588802 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:40.588782    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0903a7b6-7261-49d0-b3fc-f839f9f8ecdf" (UID: "0903a7b6-7261-49d0-b3fc-f839f9f8ecdf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:07:40.588882 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:40.588797    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-kube-api-access-kdvn6" (OuterVolumeSpecName: "kube-api-access-kdvn6") pod "0903a7b6-7261-49d0-b3fc-f839f9f8ecdf" (UID: "0903a7b6-7261-49d0-b3fc-f839f9f8ecdf"). InnerVolumeSpecName "kube-api-access-kdvn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:07:40.588920 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:40.588893    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-dshm" (OuterVolumeSpecName: "dshm") pod "0903a7b6-7261-49d0-b3fc-f839f9f8ecdf" (UID: "0903a7b6-7261-49d0-b3fc-f839f9f8ecdf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:07:40.687707 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:40.687682    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-tls-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:40.687707 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:40.687705    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-dshm\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:40.687867 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:40.687715    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdvn6\" (UniqueName: \"kubernetes.io/projected/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf-kube-api-access-kdvn6\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:07:41.339101 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:41.339067    2568 generic.go:356] "Generic (PLEG): container finished" podID="0903a7b6-7261-49d0-b3fc-f839f9f8ecdf" containerID="39b7eaa1504230d58c0ecb519c66a56ececde15e37609e4ad013a86d459b66ff" exitCode=0
Sep 11 15:07:41.339517 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:41.339103    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"
Sep 11 15:07:41.339517 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:41.339156    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5" event={"ID":"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf","Type":"ContainerDied","Data":"39b7eaa1504230d58c0ecb519c66a56ececde15e37609e4ad013a86d459b66ff"}
Sep 11 15:07:41.339517 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:41.339192    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5" event={"ID":"0903a7b6-7261-49d0-b3fc-f839f9f8ecdf","Type":"ContainerDied","Data":"b4703416bc414129d0410f4712671e2b1286e03a7af1e1b4c74b8c2b1d45e2a6"}
Sep 11 15:07:41.339517 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:41.339208    2568 scope.go:117] "RemoveContainer" containerID="39b7eaa1504230d58c0ecb519c66a56ececde15e37609e4ad013a86d459b66ff"
Sep 11 15:07:41.347805 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:41.347786    2568 scope.go:117] "RemoveContainer" containerID="39b7eaa1504230d58c0ecb519c66a56ececde15e37609e4ad013a86d459b66ff"
Sep 11 15:07:41.348071 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:07:41.348051    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b7eaa1504230d58c0ecb519c66a56ececde15e37609e4ad013a86d459b66ff\": container with ID starting with 39b7eaa1504230d58c0ecb519c66a56ececde15e37609e4ad013a86d459b66ff not found: ID does not exist" containerID="39b7eaa1504230d58c0ecb519c66a56ececde15e37609e4ad013a86d459b66ff"
Sep 11 15:07:41.348142 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:41.348077    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b7eaa1504230d58c0ecb519c66a56ececde15e37609e4ad013a86d459b66ff"} err="failed to get container status \"39b7eaa1504230d58c0ecb519c66a56ececde15e37609e4ad013a86d459b66ff\": rpc error: code = NotFound desc = could not find container \"39b7eaa1504230d58c0ecb519c66a56ececde15e37609e4ad013a86d459b66ff\": container with ID starting with 39b7eaa1504230d58c0ecb519c66a56ececde15e37609e4ad013a86d459b66ff not found: ID does not exist"
Sep 11 15:07:41.357198 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:41.357176    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"]
Sep 11 15:07:41.359594 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:41.359574    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-48898f/auth-test-model-48898f-kserve-86b799c59f-8tgv5"]
Sep 11 15:07:42.662165 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:42.662133    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0903a7b6-7261-49d0-b3fc-f839f9f8ecdf" path="/var/lib/kubelet/pods/0903a7b6-7261-49d0-b3fc-f839f9f8ecdf/volumes"
Sep 11 15:07:45.279723 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:45.279684    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:45.280137 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:45.279754    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:45.292584 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:45.292558    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:07:45.369899 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:07:45.369869    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:09:00.864921 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:00.864882    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"]
Sep 11 15:09:15.883716 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:15.883658    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9" podUID="3a6ab11c-425d-4d1a-9a7b-b18b997bde11" containerName="main" containerID="cri-o://741e917bb615874bbf0360c41b13b2ebf79930f4553457705a4b0fdf84f9e22c" gracePeriod=45
Sep 11 15:09:16.128568 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.128544    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:09:16.138463 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.138409    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-home\") pod \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") "
Sep 11 15:09:16.138463 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.138438    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-model-cache\") pod \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") "
Sep 11 15:09:16.138463 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.138464    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-tmp-dir\") pod \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") "
Sep 11 15:09:16.138669 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.138501    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-dshm\") pod \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") "
Sep 11 15:09:16.138669 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.138519    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-tls-certs\") pod \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") "
Sep 11 15:09:16.138669 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.138563    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t87hj\" (UniqueName: \"kubernetes.io/projected/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-kube-api-access-t87hj\") pod \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\" (UID: \"3a6ab11c-425d-4d1a-9a7b-b18b997bde11\") "
Sep 11 15:09:16.138848 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.138665    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a6ab11c-425d-4d1a-9a7b-b18b997bde11" (UID: "3a6ab11c-425d-4d1a-9a7b-b18b997bde11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:09:16.138848 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.138680    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-home" (OuterVolumeSpecName: "home") pod "3a6ab11c-425d-4d1a-9a7b-b18b997bde11" (UID: "3a6ab11c-425d-4d1a-9a7b-b18b997bde11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:09:16.138848 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.138798    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a6ab11c-425d-4d1a-9a7b-b18b997bde11" (UID: "3a6ab11c-425d-4d1a-9a7b-b18b997bde11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:09:16.138971 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.138877    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-home\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:09:16.138971 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.138902    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-model-cache\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:09:16.140638 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.140608    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a6ab11c-425d-4d1a-9a7b-b18b997bde11" (UID: "3a6ab11c-425d-4d1a-9a7b-b18b997bde11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:09:16.140734 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.140654    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-dshm" (OuterVolumeSpecName: "dshm") pod "3a6ab11c-425d-4d1a-9a7b-b18b997bde11" (UID: "3a6ab11c-425d-4d1a-9a7b-b18b997bde11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:09:16.140734 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.140671    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-kube-api-access-t87hj" (OuterVolumeSpecName: "kube-api-access-t87hj") pod "3a6ab11c-425d-4d1a-9a7b-b18b997bde11" (UID: "3a6ab11c-425d-4d1a-9a7b-b18b997bde11"). InnerVolumeSpecName "kube-api-access-t87hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:09:16.239496 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.239387    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-tmp-dir\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:09:16.239496 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.239420    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-dshm\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:09:16.239496 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.239431    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-tls-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:09:16.239496 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.239443    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t87hj\" (UniqueName: \"kubernetes.io/projected/3a6ab11c-425d-4d1a-9a7b-b18b997bde11-kube-api-access-t87hj\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:09:16.812751 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.812716    2568 generic.go:356] "Generic (PLEG): container finished" podID="3a6ab11c-425d-4d1a-9a7b-b18b997bde11" containerID="741e917bb615874bbf0360c41b13b2ebf79930f4553457705a4b0fdf84f9e22c" exitCode=0
Sep 11 15:09:16.812953 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.812760    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"
Sep 11 15:09:16.812953 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.812811    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9" event={"ID":"3a6ab11c-425d-4d1a-9a7b-b18b997bde11","Type":"ContainerDied","Data":"741e917bb615874bbf0360c41b13b2ebf79930f4553457705a4b0fdf84f9e22c"}
Sep 11 15:09:16.812953 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.812881    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9" event={"ID":"3a6ab11c-425d-4d1a-9a7b-b18b997bde11","Type":"ContainerDied","Data":"3a88bb88690cb8fd789ca12637c7110f68e51c4c146eed9e69e6775e30e2253b"}
Sep 11 15:09:16.812953 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.812905    2568 scope.go:117] "RemoveContainer" containerID="741e917bb615874bbf0360c41b13b2ebf79930f4553457705a4b0fdf84f9e22c"
Sep 11 15:09:16.822393 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.822368    2568 scope.go:117] "RemoveContainer" containerID="741e917bb615874bbf0360c41b13b2ebf79930f4553457705a4b0fdf84f9e22c"
Sep 11 15:09:16.826719 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:09:16.826681    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"741e917bb615874bbf0360c41b13b2ebf79930f4553457705a4b0fdf84f9e22c\": container with ID starting with 741e917bb615874bbf0360c41b13b2ebf79930f4553457705a4b0fdf84f9e22c not found: ID does not exist" containerID="741e917bb615874bbf0360c41b13b2ebf79930f4553457705a4b0fdf84f9e22c"
Sep 11 15:09:16.826933 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.826722    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741e917bb615874bbf0360c41b13b2ebf79930f4553457705a4b0fdf84f9e22c"} err="failed to get container status \"741e917bb615874bbf0360c41b13b2ebf79930f4553457705a4b0fdf84f9e22c\": rpc error: code = NotFound desc = could not find container \"741e917bb615874bbf0360c41b13b2ebf79930f4553457705a4b0fdf84f9e22c\": container with ID starting with 741e917bb615874bbf0360c41b13b2ebf79930f4553457705a4b0fdf84f9e22c not found: ID does not exist"
Sep 11 15:09:16.830711 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.830690    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"]
Sep 11 15:09:16.833363 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:16.833338    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-48898f/sub-test-model-48898f-kserve-75ff7fb6b9-xh4m9"]
Sep 11 15:09:18.663093 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:18.663058    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a6ab11c-425d-4d1a-9a7b-b18b997bde11" path="/var/lib/kubelet/pods/3a6ab11c-425d-4d1a-9a7b-b18b997bde11/volumes"
Sep 11 15:09:34.489988 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.489951    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"]
Sep 11 15:09:34.490473 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.490381    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51e90696-abd3-4ccd-b528-df0eea3b2023" containerName="istio-proxy"
Sep 11 15:09:34.490473 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.490395    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="51e90696-abd3-4ccd-b528-df0eea3b2023" containerName="istio-proxy"
Sep 11 15:09:34.490473 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.490406    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0903a7b6-7261-49d0-b3fc-f839f9f8ecdf" containerName="main"
Sep 11 15:09:34.490473 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.490411    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0903a7b6-7261-49d0-b3fc-f839f9f8ecdf" containerName="main"
Sep 11 15:09:34.490473 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.490418    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a6ab11c-425d-4d1a-9a7b-b18b997bde11" containerName="main"
Sep 11 15:09:34.490473 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.490424    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a6ab11c-425d-4d1a-9a7b-b18b997bde11" containerName="main"
Sep 11 15:09:34.490687 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.490488    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0903a7b6-7261-49d0-b3fc-f839f9f8ecdf" containerName="main"
Sep 11 15:09:34.490687 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.490505    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a6ab11c-425d-4d1a-9a7b-b18b997bde11" containerName="main"
Sep 11 15:09:34.493667 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.493648    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:34.498543 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.498519    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-928943\"/\"default-dockercfg-t992d\""
Sep 11 15:09:34.498678 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.498641    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-928943\"/\"kube-root-ca.crt\""
Sep 11 15:09:34.498678 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.498647    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-928943\"/\"rate-test-model-928943-kserve-self-signed-certs\""
Sep 11 15:09:34.503005 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.502988    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-928943\"/\"openshift-service-ca.crt\""
Sep 11 15:09:34.510423 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.510402    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"]
Sep 11 15:09:34.594527 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.594500    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-home\") pod \"rate-test-model-928943-kserve-6965996bc-jsthz\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") " pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:34.594696 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.594534    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-model-cache\") pod \"rate-test-model-928943-kserve-6965996bc-jsthz\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") " pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:34.594696 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.594554    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5396a55-2379-4176-9162-966ef5948a81-tls-certs\") pod \"rate-test-model-928943-kserve-6965996bc-jsthz\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") " pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:34.594696 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.594572    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-tmp-dir\") pod \"rate-test-model-928943-kserve-6965996bc-jsthz\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") " pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:34.594696 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.594640    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwhbl\" (UniqueName: \"kubernetes.io/projected/b5396a55-2379-4176-9162-966ef5948a81-kube-api-access-wwhbl\") pod \"rate-test-model-928943-kserve-6965996bc-jsthz\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") " pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:34.594883 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.594702    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-dshm\") pod \"rate-test-model-928943-kserve-6965996bc-jsthz\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") " pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:34.695968 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.695934    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-home\") pod \"rate-test-model-928943-kserve-6965996bc-jsthz\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") " pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:34.696167 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.695979    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-model-cache\") pod \"rate-test-model-928943-kserve-6965996bc-jsthz\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") " pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:34.696167 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.696007    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5396a55-2379-4176-9162-966ef5948a81-tls-certs\") pod \"rate-test-model-928943-kserve-6965996bc-jsthz\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") " pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:34.696167 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.696029    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-tmp-dir\") pod \"rate-test-model-928943-kserve-6965996bc-jsthz\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") " pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:34.696167 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.696070    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwhbl\" (UniqueName: \"kubernetes.io/projected/b5396a55-2379-4176-9162-966ef5948a81-kube-api-access-wwhbl\") pod \"rate-test-model-928943-kserve-6965996bc-jsthz\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") " pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:34.696400 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.696179    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-dshm\") pod \"rate-test-model-928943-kserve-6965996bc-jsthz\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") " pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:34.696400 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.696326    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-home\") pod \"rate-test-model-928943-kserve-6965996bc-jsthz\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") " pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:34.696400 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.696377    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-model-cache\") pod \"rate-test-model-928943-kserve-6965996bc-jsthz\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") " pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:34.696564 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.696434    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-tmp-dir\") pod \"rate-test-model-928943-kserve-6965996bc-jsthz\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") " pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:34.698267 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.698247    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-dshm\") pod \"rate-test-model-928943-kserve-6965996bc-jsthz\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") " pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:34.698553 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.698536    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5396a55-2379-4176-9162-966ef5948a81-tls-certs\") pod \"rate-test-model-928943-kserve-6965996bc-jsthz\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") " pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:34.704630 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.704612    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwhbl\" (UniqueName: \"kubernetes.io/projected/b5396a55-2379-4176-9162-966ef5948a81-kube-api-access-wwhbl\") pod \"rate-test-model-928943-kserve-6965996bc-jsthz\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") " pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:34.802452 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:34.802423    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:35.074153 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:35.074129    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"]
Sep 11 15:09:35.076633 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:09:35.076608    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5396a55_2379_4176_9162_966ef5948a81.slice/crio-9fc65b2e000fb6bd3d6e2b34b14bba53ba88178ee1220b48e4c2fed893701c3c WatchSource:0}: Error finding container 9fc65b2e000fb6bd3d6e2b34b14bba53ba88178ee1220b48e4c2fed893701c3c: Status 404 returned error can't find the container with id 9fc65b2e000fb6bd3d6e2b34b14bba53ba88178ee1220b48e4c2fed893701c3c
Sep 11 15:09:35.906968 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:35.906930    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz" event={"ID":"b5396a55-2379-4176-9162-966ef5948a81","Type":"ContainerStarted","Data":"9dec6d0569f10f795aba2984424af84283e69cdd322678a79ddc4cbf9d8cfe59"}
Sep 11 15:09:35.906968 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:35.906974    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz" event={"ID":"b5396a55-2379-4176-9162-966ef5948a81","Type":"ContainerStarted","Data":"9fc65b2e000fb6bd3d6e2b34b14bba53ba88178ee1220b48e4c2fed893701c3c"}
Sep 11 15:09:35.925554 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:35.925508    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz" podStartSLOduration=1.925493813 podStartE2EDuration="1.925493813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:35.924219461 +0000 UTC m=+1755.712483759" watchObservedRunningTime="2026-09-11 15:09:35.925493813 +0000 UTC m=+1755.713758113"
Sep 11 15:09:44.803296 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:44.803259    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:44.803660 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:44.803308    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:44.815619 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:44.815594    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:09:44.962258 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:09:44.962225    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:10:10.091282 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:10.091250    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"]
Sep 11 15:10:10.897032 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:10.896996    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"]
Sep 11 15:10:10.900697 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:10.900678    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:10.903056 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:10.903037    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-177c81-openshift-default-dockercfg-v82fg\""
Sep 11 15:10:10.913541 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:10.913519    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"]
Sep 11 15:10:10.922101 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:10.922069    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-podinfo\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:10.922197 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:10.922120    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-workload-socket\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:10.922247 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:10.922208    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jg8d\" (UniqueName: \"kubernetes.io/projected/46dd338f-8a78-40d4-b447-94b8b63f92b0-kube-api-access-6jg8d\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:10.922291 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:10.922251    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-token\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:10.922375 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:10.922352    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-workload-certs\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:10.922439 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:10.922396    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-data\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:10.922543 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:10.922516    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-credential-socket\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:10.922593 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:10.922576    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-envoy\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:10.922637 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:10.922608    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46dd338f-8a78-40d4-b447-94b8b63f92b0-istiod-ca-cert\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:11.023330 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.023291    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-envoy\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:11.023330 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.023329    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46dd338f-8a78-40d4-b447-94b8b63f92b0-istiod-ca-cert\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:11.023583 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.023349    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-podinfo\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:11.023583 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.023371    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-workload-socket\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:11.023583 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.023420    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jg8d\" (UniqueName: \"kubernetes.io/projected/46dd338f-8a78-40d4-b447-94b8b63f92b0-kube-api-access-6jg8d\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:11.023583 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.023446    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-token\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:11.023583 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.023484    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-workload-certs\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:11.023583 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.023511    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-data\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:11.023583 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.023576    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-credential-socket\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:11.023990 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.023921    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-workload-socket\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:11.023990 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.023957    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-workload-certs\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:11.024096 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.024067    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-data\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:11.024155 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.024131    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46dd338f-8a78-40d4-b447-94b8b63f92b0-istiod-ca-cert\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:11.024215 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.024155    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-credential-socket\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:11.025783 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.025743    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-envoy\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:11.025915 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.025884    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-podinfo\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:11.033312 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.033289    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jg8d\" (UniqueName: \"kubernetes.io/projected/46dd338f-8a78-40d4-b447-94b8b63f92b0-kube-api-access-6jg8d\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:11.033439 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.033401    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-token\") pod \"e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") " pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:11.212640 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.212543    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:11.491498 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.491398    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"]
Sep 11 15:10:11.496221 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:10:11.496187    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46dd338f_8a78_40d4_b447_94b8b63f92b0.slice/crio-e7a0cf6bf58e6ecd81ed8a9c31f7f54827fee9a6afcd7d2391dd397e05cfdc67 WatchSource:0}: Error finding container e7a0cf6bf58e6ecd81ed8a9c31f7f54827fee9a6afcd7d2391dd397e05cfdc67: Status 404 returned error can't find the container with id e7a0cf6bf58e6ecd81ed8a9c31f7f54827fee9a6afcd7d2391dd397e05cfdc67
Sep 11 15:10:11.498600 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.498563    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:10:11.498708 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.498630    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:10:11.498708 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:11.498658    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:10:12.090227 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:12.090195    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l" event={"ID":"46dd338f-8a78-40d4-b447-94b8b63f92b0","Type":"ContainerStarted","Data":"9a0d624f57698601d2ac5f651e7e5aa1f4296ab1ad69a62175480ca5bfaa9128"}
Sep 11 15:10:12.090388 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:12.090245    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l" event={"ID":"46dd338f-8a78-40d4-b447-94b8b63f92b0","Type":"ContainerStarted","Data":"e7a0cf6bf58e6ecd81ed8a9c31f7f54827fee9a6afcd7d2391dd397e05cfdc67"}
Sep 11 15:10:12.113022 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:12.112969    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l" podStartSLOduration=2.112954483 podStartE2EDuration="2.112954483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:12.111115303 +0000 UTC m=+1791.899379603" watchObservedRunningTime="2026-09-11 15:10:12.112954483 +0000 UTC m=+1791.901218782"
Sep 11 15:10:12.213202 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:12.213157    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:12.218553 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:12.218526    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:13.095561 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:13.095477    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:13.096344 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:13.096324    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:16.179681 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.179641    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"]
Sep 11 15:10:16.183688 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.183667    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"
Sep 11 15:10:16.187886 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.187852    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-177c81-serving-cert\""
Sep 11 15:10:16.193715 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.193687    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"]
Sep 11 15:10:16.276290 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.276254    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/871ed43a-6248-4376-a5e4-a7d43b7db351-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d\" (UID: \"871ed43a-6248-4376-a5e4-a7d43b7db351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"
Sep 11 15:10:16.276467 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.276307    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcmxs\" (UniqueName: \"kubernetes.io/projected/871ed43a-6248-4376-a5e4-a7d43b7db351-kube-api-access-zcmxs\") pod \"maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d\" (UID: \"871ed43a-6248-4376-a5e4-a7d43b7db351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"
Sep 11 15:10:16.276467 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.276330    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871ed43a-6248-4376-a5e4-a7d43b7db351-maas-api-tls\") pod \"maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d\" (UID: \"871ed43a-6248-4376-a5e4-a7d43b7db351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"
Sep 11 15:10:16.276467 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.276361    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871ed43a-6248-4376-a5e4-a7d43b7db351-metrics-certs\") pod \"maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d\" (UID: \"871ed43a-6248-4376-a5e4-a7d43b7db351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"
Sep 11 15:10:16.377574 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.377532    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcmxs\" (UniqueName: \"kubernetes.io/projected/871ed43a-6248-4376-a5e4-a7d43b7db351-kube-api-access-zcmxs\") pod \"maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d\" (UID: \"871ed43a-6248-4376-a5e4-a7d43b7db351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"
Sep 11 15:10:16.377574 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.377574    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871ed43a-6248-4376-a5e4-a7d43b7db351-maas-api-tls\") pod \"maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d\" (UID: \"871ed43a-6248-4376-a5e4-a7d43b7db351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"
Sep 11 15:10:16.377860 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.377594    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871ed43a-6248-4376-a5e4-a7d43b7db351-metrics-certs\") pod \"maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d\" (UID: \"871ed43a-6248-4376-a5e4-a7d43b7db351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"
Sep 11 15:10:16.377860 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.377657    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/871ed43a-6248-4376-a5e4-a7d43b7db351-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d\" (UID: \"871ed43a-6248-4376-a5e4-a7d43b7db351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"
Sep 11 15:10:16.377860 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:10:16.377702    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-serving-cert: secret "maas-api-e2e-ipp-a-177c81-serving-cert" not found
Sep 11 15:10:16.377860 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:10:16.377785    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/871ed43a-6248-4376-a5e4-a7d43b7db351-maas-api-tls podName:871ed43a-6248-4376-a5e4-a7d43b7db351 nodeName:}" failed. No retries permitted until 2026-09-11 15:10:16.877757839 +0000 UTC m=+1796.666022132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/871ed43a-6248-4376-a5e4-a7d43b7db351-maas-api-tls") pod "maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d" (UID: "871ed43a-6248-4376-a5e4-a7d43b7db351") : secret "maas-api-e2e-ipp-a-177c81-serving-cert" not found
Sep 11 15:10:16.378295 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.378272    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/871ed43a-6248-4376-a5e4-a7d43b7db351-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d\" (UID: \"871ed43a-6248-4376-a5e4-a7d43b7db351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"
Sep 11 15:10:16.380058 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.380037    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871ed43a-6248-4376-a5e4-a7d43b7db351-metrics-certs\") pod \"maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d\" (UID: \"871ed43a-6248-4376-a5e4-a7d43b7db351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"
Sep 11 15:10:16.385902 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.385880    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcmxs\" (UniqueName: \"kubernetes.io/projected/871ed43a-6248-4376-a5e4-a7d43b7db351-kube-api-access-zcmxs\") pod \"maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d\" (UID: \"871ed43a-6248-4376-a5e4-a7d43b7db351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"
Sep 11 15:10:16.882994 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.882962    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871ed43a-6248-4376-a5e4-a7d43b7db351-maas-api-tls\") pod \"maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d\" (UID: \"871ed43a-6248-4376-a5e4-a7d43b7db351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"
Sep 11 15:10:16.885379 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.885353    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871ed43a-6248-4376-a5e4-a7d43b7db351-maas-api-tls\") pod \"maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d\" (UID: \"871ed43a-6248-4376-a5e4-a7d43b7db351\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"
Sep 11 15:10:16.918697 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.918660    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j"]
Sep 11 15:10:16.922475 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.922452    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j"
Sep 11 15:10:16.924941 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.924919    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-177c81-dockercfg-lhgdl\""
Sep 11 15:10:16.925053 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.924939    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-177c81\""
Sep 11 15:10:16.932987 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.932962    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j"]
Sep 11 15:10:16.984407 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.984368    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cgxc\" (UniqueName: \"kubernetes.io/projected/20f43700-b56e-4dff-8fd7-d1d3dc563fbd-kube-api-access-2cgxc\") pod \"payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j\" (UID: \"20f43700-b56e-4dff-8fd7-d1d3dc563fbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j"
Sep 11 15:10:16.984584 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:16.984412    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f43700-b56e-4dff-8fd7-d1d3dc563fbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j\" (UID: \"20f43700-b56e-4dff-8fd7-d1d3dc563fbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j"
Sep 11 15:10:17.085762 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:17.085719    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cgxc\" (UniqueName: \"kubernetes.io/projected/20f43700-b56e-4dff-8fd7-d1d3dc563fbd-kube-api-access-2cgxc\") pod \"payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j\" (UID: \"20f43700-b56e-4dff-8fd7-d1d3dc563fbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j"
Sep 11 15:10:17.085958 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:17.085797    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f43700-b56e-4dff-8fd7-d1d3dc563fbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j\" (UID: \"20f43700-b56e-4dff-8fd7-d1d3dc563fbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j"
Sep 11 15:10:17.086495 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:17.086475    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f43700-b56e-4dff-8fd7-d1d3dc563fbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j\" (UID: \"20f43700-b56e-4dff-8fd7-d1d3dc563fbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j"
Sep 11 15:10:17.094544 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:17.094516    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cgxc\" (UniqueName: \"kubernetes.io/projected/20f43700-b56e-4dff-8fd7-d1d3dc563fbd-kube-api-access-2cgxc\") pod \"payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j\" (UID: \"20f43700-b56e-4dff-8fd7-d1d3dc563fbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j"
Sep 11 15:10:17.097387 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:17.097366    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"
Sep 11 15:10:17.234211 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:17.234175    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j"
Sep 11 15:10:17.374197 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:17.374166    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"]
Sep 11 15:10:17.375329 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:10:17.375303    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod871ed43a_6248_4376_a5e4_a7d43b7db351.slice/crio-66fc69c0b1c65b74c94061c12274773fc31de0ce851bac89445fc766b9f2a94d WatchSource:0}: Error finding container 66fc69c0b1c65b74c94061c12274773fc31de0ce851bac89445fc766b9f2a94d: Status 404 returned error can't find the container with id 66fc69c0b1c65b74c94061c12274773fc31de0ce851bac89445fc766b9f2a94d
Sep 11 15:10:17.376779 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:17.376759    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 15:10:17.504783 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:17.504700    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j"]
Sep 11 15:10:17.508390 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:10:17.508356    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f43700_b56e_4dff_8fd7_d1d3dc563fbd.slice/crio-34e748d225c756ddf164c6fac9d7dc34c0143b32121df9fd2b33180880fcc7d9 WatchSource:0}: Error finding container 34e748d225c756ddf164c6fac9d7dc34c0143b32121df9fd2b33180880fcc7d9: Status 404 returned error can't find the container with id 34e748d225c756ddf164c6fac9d7dc34c0143b32121df9fd2b33180880fcc7d9
Sep 11 15:10:18.121934 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:18.121897    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d" event={"ID":"871ed43a-6248-4376-a5e4-a7d43b7db351","Type":"ContainerStarted","Data":"7264fbbe73c6a5d5d527217d97e388816b688bcb9879b5d5bfb297ab0f751dae"}
Sep 11 15:10:18.122124 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:18.121953    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d" event={"ID":"871ed43a-6248-4376-a5e4-a7d43b7db351","Type":"ContainerStarted","Data":"66fc69c0b1c65b74c94061c12274773fc31de0ce851bac89445fc766b9f2a94d"}
Sep 11 15:10:18.122124 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:18.121981    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"
Sep 11 15:10:18.123593 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:18.123563    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j" event={"ID":"20f43700-b56e-4dff-8fd7-d1d3dc563fbd","Type":"ContainerStarted","Data":"89ef67171203b3165a10da01e1216e1d125947c63a7cf4e10cf5714144d2071b"}
Sep 11 15:10:18.123723 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:18.123607    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j" event={"ID":"20f43700-b56e-4dff-8fd7-d1d3dc563fbd","Type":"ContainerStarted","Data":"34e748d225c756ddf164c6fac9d7dc34c0143b32121df9fd2b33180880fcc7d9"}
Sep 11 15:10:18.123723 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:18.123635    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j"
Sep 11 15:10:18.142302 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:18.142252    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d" podStartSLOduration=1.522248617 podStartE2EDuration="2.142237237s" totalImagesPullingTime="619.98862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:18.141316758 +0000 UTC m=+1797.929581058" watchObservedRunningTime="2026-09-11 15:10:18.142237237 +0000 UTC m=+1797.930501536"
Sep 11 15:10:18.158185 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:18.158137    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j" podStartSLOduration=2.158123609 podStartE2EDuration="2.158123609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:18.156726201 +0000 UTC m=+1797.944990502" watchObservedRunningTime="2026-09-11 15:10:18.158123609 +0000 UTC m=+1797.946387908"
Sep 11 15:10:24.138644 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:24.138612    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"
Sep 11 15:10:25.109759 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:25.109708    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz" podUID="b5396a55-2379-4176-9162-966ef5948a81" containerName="main" containerID="cri-o://9dec6d0569f10f795aba2984424af84283e69cdd322678a79ddc4cbf9d8cfe59" gracePeriod=45
Sep 11 15:10:25.395198 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:25.395171    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:10:25.471371 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:25.471341    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-tmp-dir\") pod \"b5396a55-2379-4176-9162-966ef5948a81\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") "
Sep 11 15:10:25.471565 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:25.471387    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-model-cache\") pod \"b5396a55-2379-4176-9162-966ef5948a81\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") "
Sep 11 15:10:25.471565 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:25.471450    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5396a55-2379-4176-9162-966ef5948a81-tls-certs\") pod \"b5396a55-2379-4176-9162-966ef5948a81\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") "
Sep 11 15:10:25.471565 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:25.471483    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-home\") pod \"b5396a55-2379-4176-9162-966ef5948a81\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") "
Sep 11 15:10:25.471565 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:25.471528    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwhbl\" (UniqueName: \"kubernetes.io/projected/b5396a55-2379-4176-9162-966ef5948a81-kube-api-access-wwhbl\") pod \"b5396a55-2379-4176-9162-966ef5948a81\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") "
Sep 11 15:10:25.471565 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:25.471556    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-dshm\") pod \"b5396a55-2379-4176-9162-966ef5948a81\" (UID: \"b5396a55-2379-4176-9162-966ef5948a81\") "
Sep 11 15:10:25.471857 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:25.471618    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5396a55-2379-4176-9162-966ef5948a81" (UID: "b5396a55-2379-4176-9162-966ef5948a81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:10:25.471857 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:25.471707    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-home" (OuterVolumeSpecName: "home") pod "b5396a55-2379-4176-9162-966ef5948a81" (UID: "b5396a55-2379-4176-9162-966ef5948a81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:10:25.471857 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:25.471752    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5396a55-2379-4176-9162-966ef5948a81" (UID: "b5396a55-2379-4176-9162-966ef5948a81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:10:25.472025 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:25.471911    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-model-cache\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:10:25.472025 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:25.471931    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-home\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:10:25.472025 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:25.471945    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-tmp-dir\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:10:25.474021 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:25.473995    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5396a55-2379-4176-9162-966ef5948a81-kube-api-access-wwhbl" (OuterVolumeSpecName: "kube-api-access-wwhbl") pod "b5396a55-2379-4176-9162-966ef5948a81" (UID: "b5396a55-2379-4176-9162-966ef5948a81"). InnerVolumeSpecName "kube-api-access-wwhbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:10:25.474151 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:25.474067    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-dshm" (OuterVolumeSpecName: "dshm") pod "b5396a55-2379-4176-9162-966ef5948a81" (UID: "b5396a55-2379-4176-9162-966ef5948a81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:10:25.474151 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:25.474085    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5396a55-2379-4176-9162-966ef5948a81-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5396a55-2379-4176-9162-966ef5948a81" (UID: "b5396a55-2379-4176-9162-966ef5948a81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:10:25.573296 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:25.573260    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwhbl\" (UniqueName: \"kubernetes.io/projected/b5396a55-2379-4176-9162-966ef5948a81-kube-api-access-wwhbl\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:10:25.573296 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:25.573284    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5396a55-2379-4176-9162-966ef5948a81-dshm\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:10:25.573296 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:25.573294    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5396a55-2379-4176-9162-966ef5948a81-tls-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:10:26.170234 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:26.170199    2568 generic.go:356] "Generic (PLEG): container finished" podID="b5396a55-2379-4176-9162-966ef5948a81" containerID="9dec6d0569f10f795aba2984424af84283e69cdd322678a79ddc4cbf9d8cfe59" exitCode=0
Sep 11 15:10:26.170419 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:26.170240    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"
Sep 11 15:10:26.170419 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:26.170278    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz" event={"ID":"b5396a55-2379-4176-9162-966ef5948a81","Type":"ContainerDied","Data":"9dec6d0569f10f795aba2984424af84283e69cdd322678a79ddc4cbf9d8cfe59"}
Sep 11 15:10:26.170419 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:26.170310    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz" event={"ID":"b5396a55-2379-4176-9162-966ef5948a81","Type":"ContainerDied","Data":"9fc65b2e000fb6bd3d6e2b34b14bba53ba88178ee1220b48e4c2fed893701c3c"}
Sep 11 15:10:26.170419 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:26.170328    2568 scope.go:117] "RemoveContainer" containerID="9dec6d0569f10f795aba2984424af84283e69cdd322678a79ddc4cbf9d8cfe59"
Sep 11 15:10:26.179799 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:26.179778    2568 scope.go:117] "RemoveContainer" containerID="9dec6d0569f10f795aba2984424af84283e69cdd322678a79ddc4cbf9d8cfe59"
Sep 11 15:10:26.180087 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:10:26.180066    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dec6d0569f10f795aba2984424af84283e69cdd322678a79ddc4cbf9d8cfe59\": container with ID starting with 9dec6d0569f10f795aba2984424af84283e69cdd322678a79ddc4cbf9d8cfe59 not found: ID does not exist" containerID="9dec6d0569f10f795aba2984424af84283e69cdd322678a79ddc4cbf9d8cfe59"
Sep 11 15:10:26.180160 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:26.180099    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dec6d0569f10f795aba2984424af84283e69cdd322678a79ddc4cbf9d8cfe59"} err="failed to get container status \"9dec6d0569f10f795aba2984424af84283e69cdd322678a79ddc4cbf9d8cfe59\": rpc error: code = NotFound desc = could not find container \"9dec6d0569f10f795aba2984424af84283e69cdd322678a79ddc4cbf9d8cfe59\": container with ID starting with 9dec6d0569f10f795aba2984424af84283e69cdd322678a79ddc4cbf9d8cfe59 not found: ID does not exist"
Sep 11 15:10:26.197849 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:26.197802    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"]
Sep 11 15:10:26.201016 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:26.200996    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-928943/rate-test-model-928943-kserve-6965996bc-jsthz"]
Sep 11 15:10:26.661567 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:26.661524    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5396a55-2379-4176-9162-966ef5948a81" path="/var/lib/kubelet/pods/b5396a55-2379-4176-9162-966ef5948a81/volumes"
Sep 11 15:10:29.132778 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:29.132751    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j"
Sep 11 15:10:29.137971 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:29.137945    2568 patch_prober.go:28] interesting pod/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 15:10:29.138081 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:29.137987    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l" podUID="46dd338f-8a78-40d4-b447-94b8b63f92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 15:10:30.197879 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:30.197788    2568 generic.go:356] "Generic (PLEG): container finished" podID="46dd338f-8a78-40d4-b447-94b8b63f92b0" containerID="9a0d624f57698601d2ac5f651e7e5aa1f4296ab1ad69a62175480ca5bfaa9128" exitCode=0
Sep 11 15:10:30.197879 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:30.197854    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l" event={"ID":"46dd338f-8a78-40d4-b447-94b8b63f92b0","Type":"ContainerDied","Data":"9a0d624f57698601d2ac5f651e7e5aa1f4296ab1ad69a62175480ca5bfaa9128"}
Sep 11 15:10:30.198421 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:30.198404    2568 scope.go:117] "RemoveContainer" containerID="9a0d624f57698601d2ac5f651e7e5aa1f4296ab1ad69a62175480ca5bfaa9128"
Sep 11 15:10:30.199489 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:30.199460    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:10:30.199543 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:30.199529    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:10:30.199580 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:30.199558    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:10:31.205083 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:31.205041    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l" event={"ID":"46dd338f-8a78-40d4-b447-94b8b63f92b0","Type":"ContainerStarted","Data":"ece4b0fb2db0fb0edafdaa7ac5d8340a53c4e2d340a54122a1bced574af05ca1"}
Sep 11 15:10:31.213266 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:31.213238    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:31.213470 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:31.213441    2568 patch_prober.go:28] interesting pod/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 11 15:10:31.213545 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:31.213509    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l" podUID="46dd338f-8a78-40d4-b447-94b8b63f92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 11 15:10:32.213617 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:32.213580    2568 patch_prober.go:28] interesting pod/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 11 15:10:32.214237 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:32.214200    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l" podUID="46dd338f-8a78-40d4-b447-94b8b63f92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 11 15:10:33.214774 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:33.213001    2568 patch_prober.go:28] interesting pod/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 11 15:10:33.214774 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:33.213079    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l" podUID="46dd338f-8a78-40d4-b447-94b8b63f92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 11 15:10:34.213875 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:34.213807    2568 patch_prober.go:28] interesting pod/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 11 15:10:34.214099 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:34.213915    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l" podUID="46dd338f-8a78-40d4-b447-94b8b63f92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 11 15:10:35.213550 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:35.213510    2568 patch_prober.go:28] interesting pod/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 11 15:10:35.214179 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:35.214145    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l" podUID="46dd338f-8a78-40d4-b447-94b8b63f92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 11 15:10:36.213067 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:36.213021    2568 patch_prober.go:28] interesting pod/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 11 15:10:36.213254 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:36.213095    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l" podUID="46dd338f-8a78-40d4-b447-94b8b63f92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 11 15:10:37.217122 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:37.217089    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:37.217591 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:37.217369    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:37.218071 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:37.218050    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:10:40.280264 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.280231    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"]
Sep 11 15:10:40.280707 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.280646    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5396a55-2379-4176-9162-966ef5948a81" containerName="main"
Sep 11 15:10:40.280707 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.280657    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5396a55-2379-4176-9162-966ef5948a81" containerName="main"
Sep 11 15:10:40.280874 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.280737    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5396a55-2379-4176-9162-966ef5948a81" containerName="main"
Sep 11 15:10:40.284061 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.284038    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.287543 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.287521    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-4ee29a-openshift-default-dockercfg-jgvpq\""
Sep 11 15:10:40.297476 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.297452    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"]
Sep 11 15:10:40.421461 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.421421    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-data\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.421635 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.421491    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-credential-socket\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.421635 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.421525    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-token\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.421635 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.421561    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-envoy\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.421742 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.421639    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x7ll\" (UniqueName: \"kubernetes.io/projected/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-kube-api-access-5x7ll\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.421742 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.421678    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-workload-socket\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.421742 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.421704    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istiod-ca-cert\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.421878 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.421753    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-podinfo\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.421878 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.421784    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-workload-certs\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.522567 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.522519    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-envoy\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.522567 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.522571    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x7ll\" (UniqueName: \"kubernetes.io/projected/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-kube-api-access-5x7ll\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.522801 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.522593    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-workload-socket\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.522801 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.522616    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istiod-ca-cert\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.522973 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.522793    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-podinfo\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.522973 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.522853    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-workload-certs\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.522973 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.522925    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-data\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.523139 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.523009    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-credential-socket\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.523139 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.523049    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-token\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.523139 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.523073    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-workload-socket\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.523295 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.523257    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-workload-certs\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.523295 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.523283    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-data\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.523430 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.523413    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-credential-socket\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.523734 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.523711    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istiod-ca-cert\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.524913 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.524894    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-envoy\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.525245 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.525227    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-podinfo\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.534201 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.534144    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-token\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.534444 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.534419    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x7ll\" (UniqueName: \"kubernetes.io/projected/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-kube-api-access-5x7ll\") pod \"e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") " pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.595091 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.595053    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:40.880165 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.880132    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"]
Sep 11 15:10:40.881621 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:10:40.881596    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38d49d7_ae58_46cd_b50c_5c103e52e5fd.slice/crio-eed1b28048c838ff9a6f81ba519468e2b4a9d3b5b556944a3eeac8fd96601947 WatchSource:0}: Error finding container eed1b28048c838ff9a6f81ba519468e2b4a9d3b5b556944a3eeac8fd96601947: Status 404 returned error can't find the container with id eed1b28048c838ff9a6f81ba519468e2b4a9d3b5b556944a3eeac8fd96601947
Sep 11 15:10:40.884317 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.884281    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:10:40.884414 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.884365    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:10:40.884479 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:40.884409    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:10:41.268561 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:41.268462    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r" event={"ID":"a38d49d7-ae58-46cd-b50c-5c103e52e5fd","Type":"ContainerStarted","Data":"b64ba9b82346b63708090e9446722f5647c9f2226c08dd1c7e12857ef81aeb00"}
Sep 11 15:10:41.268794 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:41.268562    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r" event={"ID":"a38d49d7-ae58-46cd-b50c-5c103e52e5fd","Type":"ContainerStarted","Data":"eed1b28048c838ff9a6f81ba519468e2b4a9d3b5b556944a3eeac8fd96601947"}
Sep 11 15:10:41.595642 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:41.595603    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:41.600532 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:41.600509    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:41.624596 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:41.624544    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r" podStartSLOduration=1.624527816 podStartE2EDuration="1.624527816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:41.306487483 +0000 UTC m=+1821.094751783" watchObservedRunningTime="2026-09-11 15:10:41.624527816 +0000 UTC m=+1821.412792115"
Sep 11 15:10:42.275139 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:42.275107    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:42.276198 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:42.276177    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:10:46.697394 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:46.697361    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t"]
Sep 11 15:10:46.702090 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:46.702071    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t"
Sep 11 15:10:46.705036 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:46.705011    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4ee29a\""
Sep 11 15:10:46.713728 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:46.713704    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t"]
Sep 11 15:10:46.888257 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:46.888224    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-746v6\" (UniqueName: \"kubernetes.io/projected/3389266a-942d-4a14-9745-7be666ed8b48-kube-api-access-746v6\") pod \"payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t\" (UID: \"3389266a-942d-4a14-9745-7be666ed8b48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t"
Sep 11 15:10:46.888437 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:46.888394    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3389266a-942d-4a14-9745-7be666ed8b48-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t\" (UID: \"3389266a-942d-4a14-9745-7be666ed8b48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t"
Sep 11 15:10:46.989439 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:46.989353    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3389266a-942d-4a14-9745-7be666ed8b48-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t\" (UID: \"3389266a-942d-4a14-9745-7be666ed8b48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t"
Sep 11 15:10:46.989439 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:46.989405    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-746v6\" (UniqueName: \"kubernetes.io/projected/3389266a-942d-4a14-9745-7be666ed8b48-kube-api-access-746v6\") pod \"payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t\" (UID: \"3389266a-942d-4a14-9745-7be666ed8b48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t"
Sep 11 15:10:46.990143 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:46.990122    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3389266a-942d-4a14-9745-7be666ed8b48-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t\" (UID: \"3389266a-942d-4a14-9745-7be666ed8b48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t"
Sep 11 15:10:46.998326 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:46.998291    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-746v6\" (UniqueName: \"kubernetes.io/projected/3389266a-942d-4a14-9745-7be666ed8b48-kube-api-access-746v6\") pod \"payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t\" (UID: \"3389266a-942d-4a14-9745-7be666ed8b48\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t"
Sep 11 15:10:47.014716 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:47.014692    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t"
Sep 11 15:10:47.289499 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:47.289443    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t"]
Sep 11 15:10:47.292753 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:10:47.292726    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3389266a_942d_4a14_9745_7be666ed8b48.slice/crio-c55646052e62d3404001703ac2f20012b52e7e9342d6513c93b9ee5551e4734d WatchSource:0}: Error finding container c55646052e62d3404001703ac2f20012b52e7e9342d6513c93b9ee5551e4734d: Status 404 returned error can't find the container with id c55646052e62d3404001703ac2f20012b52e7e9342d6513c93b9ee5551e4734d
Sep 11 15:10:47.301613 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:47.301586    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t" event={"ID":"3389266a-942d-4a14-9745-7be666ed8b48","Type":"ContainerStarted","Data":"c55646052e62d3404001703ac2f20012b52e7e9342d6513c93b9ee5551e4734d"}
Sep 11 15:10:48.308850 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:48.308793    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t" event={"ID":"3389266a-942d-4a14-9745-7be666ed8b48","Type":"ContainerStarted","Data":"805fe0bcdd22e8f1e50f3a9ccca8a1d90d5d40889927829029f7d8b8f1f7d615"}
Sep 11 15:10:48.309277 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:48.308885    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t"
Sep 11 15:10:48.327477 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:48.327424    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t" podStartSLOduration=2.327410284 podStartE2EDuration="2.327410284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:48.32610118 +0000 UTC m=+1828.114365504" watchObservedRunningTime="2026-09-11 15:10:48.327410284 +0000 UTC m=+1828.115674582"
Sep 11 15:10:59.316436 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:10:59.316403    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t"
Sep 11 15:11:01.382894 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:01.382788    2568 generic.go:356] "Generic (PLEG): container finished" podID="a38d49d7-ae58-46cd-b50c-5c103e52e5fd" containerID="b64ba9b82346b63708090e9446722f5647c9f2226c08dd1c7e12857ef81aeb00" exitCode=0
Sep 11 15:11:01.382894 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:01.382866    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r" event={"ID":"a38d49d7-ae58-46cd-b50c-5c103e52e5fd","Type":"ContainerDied","Data":"b64ba9b82346b63708090e9446722f5647c9f2226c08dd1c7e12857ef81aeb00"}
Sep 11 15:11:01.383493 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:01.383404    2568 scope.go:117] "RemoveContainer" containerID="b64ba9b82346b63708090e9446722f5647c9f2226c08dd1c7e12857ef81aeb00"
Sep 11 15:11:01.384523 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:01.384495    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:11:01.384583 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:01.384571    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:11:01.384630 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:01.384604    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 15:11:02.390254 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:02.390218    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r" event={"ID":"a38d49d7-ae58-46cd-b50c-5c103e52e5fd","Type":"ContainerStarted","Data":"f3f6408b2afdd3e442c5b4ecabbf603e39c1c3f6649cc97479e75926f3a51769"}
Sep 11 15:11:02.596001 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:02.595959    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:11:02.596229 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:02.596203    2568 patch_prober.go:28] interesting pod/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 11 15:11:02.596319 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:02.596277    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r" podUID="a38d49d7-ae58-46cd-b50c-5c103e52e5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 11 15:11:03.595505 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:03.595467    2568 patch_prober.go:28] interesting pod/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 11 15:11:03.596061 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:03.595535    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r" podUID="a38d49d7-ae58-46cd-b50c-5c103e52e5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 11 15:11:04.596424 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:04.596380    2568 patch_prober.go:28] interesting pod/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 11 15:11:04.596957 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:04.596445    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r" podUID="a38d49d7-ae58-46cd-b50c-5c103e52e5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 11 15:11:05.596065 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:05.596025    2568 patch_prober.go:28] interesting pod/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 11 15:11:05.596244 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:05.596082    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r" podUID="a38d49d7-ae58-46cd-b50c-5c103e52e5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 11 15:11:06.595715 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:06.595680    2568 patch_prober.go:28] interesting pod/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 11 15:11:06.596133 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:06.595734    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r" podUID="a38d49d7-ae58-46cd-b50c-5c103e52e5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 11 15:11:07.677235 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:07.677205    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:11:07.677625 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:07.677473    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:11:07.678160 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:07.678141    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:11:13.587236 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:11:13.587205    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-rwvw4_0bc37496-0e43-4d25-a663-a96dd2e1f64e/payload-processing/0.log"
Sep 11 15:12:05.360337 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:05.360302    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-rwvw4_0bc37496-0e43-4d25-a663-a96dd2e1f64e/payload-processing/0.log"
Sep 11 15:12:12.979073 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:12.979035    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"]
Sep 11 15:12:12.983547 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:12.983524    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"
Sep 11 15:12:12.987172 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:12.987146    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-e04dee-serving-cert\""
Sep 11 15:12:12.998486 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:12.998460    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"]
Sep 11 15:12:13.035430 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.035394    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a0f069f-98b9-45d8-9058-1e582f39e219-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77\" (UID: \"8a0f069f-98b9-45d8-9058-1e582f39e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"
Sep 11 15:12:13.035637 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.035447    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a0f069f-98b9-45d8-9058-1e582f39e219-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77\" (UID: \"8a0f069f-98b9-45d8-9058-1e582f39e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"
Sep 11 15:12:13.035637 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.035529    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a0f069f-98b9-45d8-9058-1e582f39e219-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77\" (UID: \"8a0f069f-98b9-45d8-9058-1e582f39e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"
Sep 11 15:12:13.035637 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.035563    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr29s\" (UniqueName: \"kubernetes.io/projected/8a0f069f-98b9-45d8-9058-1e582f39e219-kube-api-access-qr29s\") pod \"maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77\" (UID: \"8a0f069f-98b9-45d8-9058-1e582f39e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"
Sep 11 15:12:13.137084 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.137031    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a0f069f-98b9-45d8-9058-1e582f39e219-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77\" (UID: \"8a0f069f-98b9-45d8-9058-1e582f39e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"
Sep 11 15:12:13.137084 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.137099    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a0f069f-98b9-45d8-9058-1e582f39e219-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77\" (UID: \"8a0f069f-98b9-45d8-9058-1e582f39e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"
Sep 11 15:12:13.137349 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.137128    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr29s\" (UniqueName: \"kubernetes.io/projected/8a0f069f-98b9-45d8-9058-1e582f39e219-kube-api-access-qr29s\") pod \"maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77\" (UID: \"8a0f069f-98b9-45d8-9058-1e582f39e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"
Sep 11 15:12:13.137349 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.137233    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a0f069f-98b9-45d8-9058-1e582f39e219-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77\" (UID: \"8a0f069f-98b9-45d8-9058-1e582f39e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"
Sep 11 15:12:13.137349 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:12:13.137276    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-serving-cert: secret "maas-api-e2e-ipp-cleanup-e04dee-serving-cert" not found
Sep 11 15:12:13.137509 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:12:13.137377    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a0f069f-98b9-45d8-9058-1e582f39e219-maas-api-tls podName:8a0f069f-98b9-45d8-9058-1e582f39e219 nodeName:}" failed. No retries permitted until 2026-09-11 15:12:13.63735663 +0000 UTC m=+1913.425620922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a0f069f-98b9-45d8-9058-1e582f39e219-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77" (UID: "8a0f069f-98b9-45d8-9058-1e582f39e219") : secret "maas-api-e2e-ipp-cleanup-e04dee-serving-cert" not found
Sep 11 15:12:13.137909 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.137891    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a0f069f-98b9-45d8-9058-1e582f39e219-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77\" (UID: \"8a0f069f-98b9-45d8-9058-1e582f39e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"
Sep 11 15:12:13.139640 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.139614    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a0f069f-98b9-45d8-9058-1e582f39e219-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77\" (UID: \"8a0f069f-98b9-45d8-9058-1e582f39e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"
Sep 11 15:12:13.146541 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.146510    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr29s\" (UniqueName: \"kubernetes.io/projected/8a0f069f-98b9-45d8-9058-1e582f39e219-kube-api-access-qr29s\") pod \"maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77\" (UID: \"8a0f069f-98b9-45d8-9058-1e582f39e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"
Sep 11 15:12:13.642758 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.642720    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a0f069f-98b9-45d8-9058-1e582f39e219-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77\" (UID: \"8a0f069f-98b9-45d8-9058-1e582f39e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"
Sep 11 15:12:13.642962 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:12:13.642904    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-serving-cert: secret "maas-api-e2e-ipp-cleanup-e04dee-serving-cert" not found
Sep 11 15:12:13.643018 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:12:13.642978    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a0f069f-98b9-45d8-9058-1e582f39e219-maas-api-tls podName:8a0f069f-98b9-45d8-9058-1e582f39e219 nodeName:}" failed. No retries permitted until 2026-09-11 15:12:14.642962009 +0000 UTC m=+1914.431226285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a0f069f-98b9-45d8-9058-1e582f39e219-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77" (UID: "8a0f069f-98b9-45d8-9058-1e582f39e219") : secret "maas-api-e2e-ipp-cleanup-e04dee-serving-cert" not found
Sep 11 15:12:13.652470 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.652439    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp"]
Sep 11 15:12:13.656567 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.656543    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp"
Sep 11 15:12:13.659268 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.659241    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-e04dee\""
Sep 11 15:12:13.659406 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.659242    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-e04dee-dockercfg-q24cg\""
Sep 11 15:12:13.668199 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.668174    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp"]
Sep 11 15:12:13.743685 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.743649    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af900bff-c3fd-4d8f-968b-962c71d078bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp\" (UID: \"af900bff-c3fd-4d8f-968b-962c71d078bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp"
Sep 11 15:12:13.743685 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.743688    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8tw4\" (UniqueName: \"kubernetes.io/projected/af900bff-c3fd-4d8f-968b-962c71d078bd-kube-api-access-h8tw4\") pod \"payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp\" (UID: \"af900bff-c3fd-4d8f-968b-962c71d078bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp"
Sep 11 15:12:13.844439 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.844397    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af900bff-c3fd-4d8f-968b-962c71d078bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp\" (UID: \"af900bff-c3fd-4d8f-968b-962c71d078bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp"
Sep 11 15:12:13.844439 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.844437    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8tw4\" (UniqueName: \"kubernetes.io/projected/af900bff-c3fd-4d8f-968b-962c71d078bd-kube-api-access-h8tw4\") pod \"payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp\" (UID: \"af900bff-c3fd-4d8f-968b-962c71d078bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp"
Sep 11 15:12:13.845165 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.845146    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af900bff-c3fd-4d8f-968b-962c71d078bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp\" (UID: \"af900bff-c3fd-4d8f-968b-962c71d078bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp"
Sep 11 15:12:13.855034 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.855006    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8tw4\" (UniqueName: \"kubernetes.io/projected/af900bff-c3fd-4d8f-968b-962c71d078bd-kube-api-access-h8tw4\") pod \"payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp\" (UID: \"af900bff-c3fd-4d8f-968b-962c71d078bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp"
Sep 11 15:12:13.969939 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:13.969840    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp"
Sep 11 15:12:14.246277 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:14.246250    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp"]
Sep 11 15:12:14.248379 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:12:14.248352    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf900bff_c3fd_4d8f_968b_962c71d078bd.slice/crio-22aeb88c2548a5e32fbc28f7f618241122dbebe59822a709ad2715b7081cc99c WatchSource:0}: Error finding container 22aeb88c2548a5e32fbc28f7f618241122dbebe59822a709ad2715b7081cc99c: Status 404 returned error can't find the container with id 22aeb88c2548a5e32fbc28f7f618241122dbebe59822a709ad2715b7081cc99c
Sep 11 15:12:14.651955 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:14.651915    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a0f069f-98b9-45d8-9058-1e582f39e219-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77\" (UID: \"8a0f069f-98b9-45d8-9058-1e582f39e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"
Sep 11 15:12:14.654512 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:14.654488    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a0f069f-98b9-45d8-9058-1e582f39e219-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77\" (UID: \"8a0f069f-98b9-45d8-9058-1e582f39e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"
Sep 11 15:12:14.765993 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:14.765950    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp" event={"ID":"af900bff-c3fd-4d8f-968b-962c71d078bd","Type":"ContainerStarted","Data":"79764eff546421582db9b085409e2c469b4cef1517027dc0e116ba94836f11a4"}
Sep 11 15:12:14.766205 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:14.766019    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp"
Sep 11 15:12:14.766205 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:14.766043    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp" event={"ID":"af900bff-c3fd-4d8f-968b-962c71d078bd","Type":"ContainerStarted","Data":"22aeb88c2548a5e32fbc28f7f618241122dbebe59822a709ad2715b7081cc99c"}
Sep 11 15:12:14.786439 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:14.786390    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp" podStartSLOduration=1.786376555 podStartE2EDuration="1.786376555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:12:14.785043716 +0000 UTC m=+1914.573308017" watchObservedRunningTime="2026-09-11 15:12:14.786376555 +0000 UTC m=+1914.574640854"
Sep 11 15:12:14.795224 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:14.795202    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"
Sep 11 15:12:15.075909 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:15.075881    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"]
Sep 11 15:12:15.077654 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:12:15.077631    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0f069f_98b9_45d8_9058_1e582f39e219.slice/crio-d82ec4a465d1850bf70392e40231dcda2161a247621b22bd9871ebd88bba074e WatchSource:0}: Error finding container d82ec4a465d1850bf70392e40231dcda2161a247621b22bd9871ebd88bba074e: Status 404 returned error can't find the container with id d82ec4a465d1850bf70392e40231dcda2161a247621b22bd9871ebd88bba074e
Sep 11 15:12:15.772377 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:15.772344    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77" event={"ID":"8a0f069f-98b9-45d8-9058-1e582f39e219","Type":"ContainerStarted","Data":"01a83757782deac4f13e1ffda677ea6eadb368f777e04d52e4ae42a435e9eb8d"}
Sep 11 15:12:15.772797 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:15.772400    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77" event={"ID":"8a0f069f-98b9-45d8-9058-1e582f39e219","Type":"ContainerStarted","Data":"d82ec4a465d1850bf70392e40231dcda2161a247621b22bd9871ebd88bba074e"}
Sep 11 15:12:15.772797 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:15.772555    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"
Sep 11 15:12:15.790296 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:15.790243    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77" podStartSLOduration=3.253435664 podStartE2EDuration="3.790226037s" totalImagesPullingTime="536.790373ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:12:15.789936039 +0000 UTC m=+1915.578200337" watchObservedRunningTime="2026-09-11 15:12:15.790226037 +0000 UTC m=+1915.578490337"
Sep 11 15:12:21.783110 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:21.783079    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"
Sep 11 15:12:25.774699 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:25.774667    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp"
Sep 11 15:12:50.779860 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:50.779809    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"]
Sep 11 15:12:50.780426 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:50.780075    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77" podUID="8a0f069f-98b9-45d8-9058-1e582f39e219" containerName="maas-api" containerID="cri-o://01a83757782deac4f13e1ffda677ea6eadb368f777e04d52e4ae42a435e9eb8d" gracePeriod=30
Sep 11 15:12:50.846509 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:50.846478    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp"]
Sep 11 15:12:50.846788 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:50.846754    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp" podUID="af900bff-c3fd-4d8f-968b-962c71d078bd" containerName="payload-processing" containerID="cri-o://79764eff546421582db9b085409e2c469b4cef1517027dc0e116ba94836f11a4" gracePeriod=30
Sep 11 15:12:50.957540 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:50.957512    2568 generic.go:356] "Generic (PLEG): container finished" podID="8a0f069f-98b9-45d8-9058-1e582f39e219" containerID="01a83757782deac4f13e1ffda677ea6eadb368f777e04d52e4ae42a435e9eb8d" exitCode=0
Sep 11 15:12:50.957639 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:50.957566    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77" event={"ID":"8a0f069f-98b9-45d8-9058-1e582f39e219","Type":"ContainerDied","Data":"01a83757782deac4f13e1ffda677ea6eadb368f777e04d52e4ae42a435e9eb8d"}
Sep 11 15:12:51.121032 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.120724    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"
Sep 11 15:12:51.154746 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.154706    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp"
Sep 11 15:12:51.205073 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.205042    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af900bff-c3fd-4d8f-968b-962c71d078bd-plugins-config-volume\") pod \"af900bff-c3fd-4d8f-968b-962c71d078bd\" (UID: \"af900bff-c3fd-4d8f-968b-962c71d078bd\") "
Sep 11 15:12:51.205235 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.205079    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a0f069f-98b9-45d8-9058-1e582f39e219-openshift-service-ca\") pod \"8a0f069f-98b9-45d8-9058-1e582f39e219\" (UID: \"8a0f069f-98b9-45d8-9058-1e582f39e219\") "
Sep 11 15:12:51.205235 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.205098    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a0f069f-98b9-45d8-9058-1e582f39e219-maas-api-tls\") pod \"8a0f069f-98b9-45d8-9058-1e582f39e219\" (UID: \"8a0f069f-98b9-45d8-9058-1e582f39e219\") "
Sep 11 15:12:51.205235 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.205178    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr29s\" (UniqueName: \"kubernetes.io/projected/8a0f069f-98b9-45d8-9058-1e582f39e219-kube-api-access-qr29s\") pod \"8a0f069f-98b9-45d8-9058-1e582f39e219\" (UID: \"8a0f069f-98b9-45d8-9058-1e582f39e219\") "
Sep 11 15:12:51.205235 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.205204    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a0f069f-98b9-45d8-9058-1e582f39e219-metrics-certs\") pod \"8a0f069f-98b9-45d8-9058-1e582f39e219\" (UID: \"8a0f069f-98b9-45d8-9058-1e582f39e219\") "
Sep 11 15:12:51.205459 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.205235    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8tw4\" (UniqueName: \"kubernetes.io/projected/af900bff-c3fd-4d8f-968b-962c71d078bd-kube-api-access-h8tw4\") pod \"af900bff-c3fd-4d8f-968b-962c71d078bd\" (UID: \"af900bff-c3fd-4d8f-968b-962c71d078bd\") "
Sep 11 15:12:51.205526 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.205476    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af900bff-c3fd-4d8f-968b-962c71d078bd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af900bff-c3fd-4d8f-968b-962c71d078bd" (UID: "af900bff-c3fd-4d8f-968b-962c71d078bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:12:51.205590 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.205565    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a0f069f-98b9-45d8-9058-1e582f39e219-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8a0f069f-98b9-45d8-9058-1e582f39e219" (UID: "8a0f069f-98b9-45d8-9058-1e582f39e219"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:12:51.205676 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.205651    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a0f069f-98b9-45d8-9058-1e582f39e219-openshift-service-ca\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:12:51.205734 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.205676    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af900bff-c3fd-4d8f-968b-962c71d078bd-plugins-config-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:12:51.207495 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.207466    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0f069f-98b9-45d8-9058-1e582f39e219-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8a0f069f-98b9-45d8-9058-1e582f39e219" (UID: "8a0f069f-98b9-45d8-9058-1e582f39e219"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:12:51.207495 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.207482    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a0f069f-98b9-45d8-9058-1e582f39e219-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8a0f069f-98b9-45d8-9058-1e582f39e219" (UID: "8a0f069f-98b9-45d8-9058-1e582f39e219"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:12:51.207649 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.207467    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0f069f-98b9-45d8-9058-1e582f39e219-kube-api-access-qr29s" (OuterVolumeSpecName: "kube-api-access-qr29s") pod "8a0f069f-98b9-45d8-9058-1e582f39e219" (UID: "8a0f069f-98b9-45d8-9058-1e582f39e219"). InnerVolumeSpecName "kube-api-access-qr29s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:12:51.207649 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.207478    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af900bff-c3fd-4d8f-968b-962c71d078bd-kube-api-access-h8tw4" (OuterVolumeSpecName: "kube-api-access-h8tw4") pod "af900bff-c3fd-4d8f-968b-962c71d078bd" (UID: "af900bff-c3fd-4d8f-968b-962c71d078bd"). InnerVolumeSpecName "kube-api-access-h8tw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:12:51.307223 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.307137    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a0f069f-98b9-45d8-9058-1e582f39e219-maas-api-tls\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:12:51.307223 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.307175    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr29s\" (UniqueName: \"kubernetes.io/projected/8a0f069f-98b9-45d8-9058-1e582f39e219-kube-api-access-qr29s\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:12:51.307223 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.307185    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a0f069f-98b9-45d8-9058-1e582f39e219-metrics-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:12:51.307223 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.307193    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8tw4\" (UniqueName: \"kubernetes.io/projected/af900bff-c3fd-4d8f-968b-962c71d078bd-kube-api-access-h8tw4\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:12:51.963804 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.963768    2568 generic.go:356] "Generic (PLEG): container finished" podID="af900bff-c3fd-4d8f-968b-962c71d078bd" containerID="79764eff546421582db9b085409e2c469b4cef1517027dc0e116ba94836f11a4" exitCode=0
Sep 11 15:12:51.964237 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.963833    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp"
Sep 11 15:12:51.964237 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.963852    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp" event={"ID":"af900bff-c3fd-4d8f-968b-962c71d078bd","Type":"ContainerDied","Data":"79764eff546421582db9b085409e2c469b4cef1517027dc0e116ba94836f11a4"}
Sep 11 15:12:51.964237 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.963906    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp" event={"ID":"af900bff-c3fd-4d8f-968b-962c71d078bd","Type":"ContainerDied","Data":"22aeb88c2548a5e32fbc28f7f618241122dbebe59822a709ad2715b7081cc99c"}
Sep 11 15:12:51.964237 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.963928    2568 scope.go:117] "RemoveContainer" containerID="79764eff546421582db9b085409e2c469b4cef1517027dc0e116ba94836f11a4"
Sep 11 15:12:51.965363 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.965344    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"
Sep 11 15:12:51.965445 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.965381    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77" event={"ID":"8a0f069f-98b9-45d8-9058-1e582f39e219","Type":"ContainerDied","Data":"d82ec4a465d1850bf70392e40231dcda2161a247621b22bd9871ebd88bba074e"}
Sep 11 15:12:51.974187 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.974171    2568 scope.go:117] "RemoveContainer" containerID="79764eff546421582db9b085409e2c469b4cef1517027dc0e116ba94836f11a4"
Sep 11 15:12:51.974428 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:12:51.974408    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79764eff546421582db9b085409e2c469b4cef1517027dc0e116ba94836f11a4\": container with ID starting with 79764eff546421582db9b085409e2c469b4cef1517027dc0e116ba94836f11a4 not found: ID does not exist" containerID="79764eff546421582db9b085409e2c469b4cef1517027dc0e116ba94836f11a4"
Sep 11 15:12:51.974492 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.974437    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79764eff546421582db9b085409e2c469b4cef1517027dc0e116ba94836f11a4"} err="failed to get container status \"79764eff546421582db9b085409e2c469b4cef1517027dc0e116ba94836f11a4\": rpc error: code = NotFound desc = could not find container \"79764eff546421582db9b085409e2c469b4cef1517027dc0e116ba94836f11a4\": container with ID starting with 79764eff546421582db9b085409e2c469b4cef1517027dc0e116ba94836f11a4 not found: ID does not exist"
Sep 11 15:12:51.974492 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.974461    2568 scope.go:117] "RemoveContainer" containerID="01a83757782deac4f13e1ffda677ea6eadb368f777e04d52e4ae42a435e9eb8d"
Sep 11 15:12:51.987966 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.987938    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp"]
Sep 11 15:12:51.990128 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:51.990108    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e04dee-546cb95c69-pwbwp"]
Sep 11 15:12:52.000233 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:52.000213    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"]
Sep 11 15:12:52.004688 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:52.004661    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e04dee-75b9d94bfb-sts77"]
Sep 11 15:12:52.661663 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:52.661627    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a0f069f-98b9-45d8-9058-1e582f39e219" path="/var/lib/kubelet/pods/8a0f069f-98b9-45d8-9058-1e582f39e219/volumes"
Sep 11 15:12:52.662044 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:12:52.662028    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af900bff-c3fd-4d8f-968b-962c71d078bd" path="/var/lib/kubelet/pods/af900bff-c3fd-4d8f-968b-962c71d078bd/volumes"
Sep 11 15:13:17.337625 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.337542    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"]
Sep 11 15:13:17.338142 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.337851    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d" podUID="871ed43a-6248-4376-a5e4-a7d43b7db351" containerName="maas-api" containerID="cri-o://7264fbbe73c6a5d5d527217d97e388816b688bcb9879b5d5bfb297ab0f751dae" gracePeriod=30
Sep 11 15:13:17.398637 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.398600    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j"]
Sep 11 15:13:17.398891 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.398843    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j" podUID="20f43700-b56e-4dff-8fd7-d1d3dc563fbd" containerName="payload-pre-processing" containerID="cri-o://89ef67171203b3165a10da01e1216e1d125947c63a7cf4e10cf5714144d2071b" gracePeriod=30
Sep 11 15:13:17.666651 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.666623    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"
Sep 11 15:13:17.693591 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.693564    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j"
Sep 11 15:13:17.744762 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.744734    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcmxs\" (UniqueName: \"kubernetes.io/projected/871ed43a-6248-4376-a5e4-a7d43b7db351-kube-api-access-zcmxs\") pod \"871ed43a-6248-4376-a5e4-a7d43b7db351\" (UID: \"871ed43a-6248-4376-a5e4-a7d43b7db351\") "
Sep 11 15:13:17.744928 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.744777    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871ed43a-6248-4376-a5e4-a7d43b7db351-metrics-certs\") pod \"871ed43a-6248-4376-a5e4-a7d43b7db351\" (UID: \"871ed43a-6248-4376-a5e4-a7d43b7db351\") "
Sep 11 15:13:17.744928 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.744859    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871ed43a-6248-4376-a5e4-a7d43b7db351-maas-api-tls\") pod \"871ed43a-6248-4376-a5e4-a7d43b7db351\" (UID: \"871ed43a-6248-4376-a5e4-a7d43b7db351\") "
Sep 11 15:13:17.745072 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.745048    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/871ed43a-6248-4376-a5e4-a7d43b7db351-openshift-service-ca\") pod \"871ed43a-6248-4376-a5e4-a7d43b7db351\" (UID: \"871ed43a-6248-4376-a5e4-a7d43b7db351\") "
Sep 11 15:13:17.745377 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.745339    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/871ed43a-6248-4376-a5e4-a7d43b7db351-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "871ed43a-6248-4376-a5e4-a7d43b7db351" (UID: "871ed43a-6248-4376-a5e4-a7d43b7db351"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:13:17.745489 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.745474    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/871ed43a-6248-4376-a5e4-a7d43b7db351-openshift-service-ca\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:13:17.746894 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.746874    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/871ed43a-6248-4376-a5e4-a7d43b7db351-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "871ed43a-6248-4376-a5e4-a7d43b7db351" (UID: "871ed43a-6248-4376-a5e4-a7d43b7db351"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:13:17.747318 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.747298    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/871ed43a-6248-4376-a5e4-a7d43b7db351-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "871ed43a-6248-4376-a5e4-a7d43b7db351" (UID: "871ed43a-6248-4376-a5e4-a7d43b7db351"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:13:17.747318 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.747307    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871ed43a-6248-4376-a5e4-a7d43b7db351-kube-api-access-zcmxs" (OuterVolumeSpecName: "kube-api-access-zcmxs") pod "871ed43a-6248-4376-a5e4-a7d43b7db351" (UID: "871ed43a-6248-4376-a5e4-a7d43b7db351"). InnerVolumeSpecName "kube-api-access-zcmxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:13:17.846129 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.846099    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cgxc\" (UniqueName: \"kubernetes.io/projected/20f43700-b56e-4dff-8fd7-d1d3dc563fbd-kube-api-access-2cgxc\") pod \"20f43700-b56e-4dff-8fd7-d1d3dc563fbd\" (UID: \"20f43700-b56e-4dff-8fd7-d1d3dc563fbd\") "
Sep 11 15:13:17.846293 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.846212    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f43700-b56e-4dff-8fd7-d1d3dc563fbd-plugins-config-volume\") pod \"20f43700-b56e-4dff-8fd7-d1d3dc563fbd\" (UID: \"20f43700-b56e-4dff-8fd7-d1d3dc563fbd\") "
Sep 11 15:13:17.846409 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.846396    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcmxs\" (UniqueName: \"kubernetes.io/projected/871ed43a-6248-4376-a5e4-a7d43b7db351-kube-api-access-zcmxs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:13:17.846471 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.846410    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871ed43a-6248-4376-a5e4-a7d43b7db351-metrics-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:13:17.846471 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.846419    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871ed43a-6248-4376-a5e4-a7d43b7db351-maas-api-tls\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:13:17.846621 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.846595    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f43700-b56e-4dff-8fd7-d1d3dc563fbd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "20f43700-b56e-4dff-8fd7-d1d3dc563fbd" (UID: "20f43700-b56e-4dff-8fd7-d1d3dc563fbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:13:17.848051 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.848030    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f43700-b56e-4dff-8fd7-d1d3dc563fbd-kube-api-access-2cgxc" (OuterVolumeSpecName: "kube-api-access-2cgxc") pod "20f43700-b56e-4dff-8fd7-d1d3dc563fbd" (UID: "20f43700-b56e-4dff-8fd7-d1d3dc563fbd"). InnerVolumeSpecName "kube-api-access-2cgxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:13:17.947393 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.947320    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f43700-b56e-4dff-8fd7-d1d3dc563fbd-plugins-config-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:13:17.947393 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:17.947350    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cgxc\" (UniqueName: \"kubernetes.io/projected/20f43700-b56e-4dff-8fd7-d1d3dc563fbd-kube-api-access-2cgxc\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:13:18.112150 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:18.112112    2568 generic.go:356] "Generic (PLEG): container finished" podID="871ed43a-6248-4376-a5e4-a7d43b7db351" containerID="7264fbbe73c6a5d5d527217d97e388816b688bcb9879b5d5bfb297ab0f751dae" exitCode=0
Sep 11 15:13:18.112349 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:18.112168    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"
Sep 11 15:13:18.112349 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:18.112191    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d" event={"ID":"871ed43a-6248-4376-a5e4-a7d43b7db351","Type":"ContainerDied","Data":"7264fbbe73c6a5d5d527217d97e388816b688bcb9879b5d5bfb297ab0f751dae"}
Sep 11 15:13:18.112349 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:18.112257    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d" event={"ID":"871ed43a-6248-4376-a5e4-a7d43b7db351","Type":"ContainerDied","Data":"66fc69c0b1c65b74c94061c12274773fc31de0ce851bac89445fc766b9f2a94d"}
Sep 11 15:13:18.112349 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:18.112289    2568 scope.go:117] "RemoveContainer" containerID="7264fbbe73c6a5d5d527217d97e388816b688bcb9879b5d5bfb297ab0f751dae"
Sep 11 15:13:18.114031 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:18.113966    2568 generic.go:356] "Generic (PLEG): container finished" podID="20f43700-b56e-4dff-8fd7-d1d3dc563fbd" containerID="89ef67171203b3165a10da01e1216e1d125947c63a7cf4e10cf5714144d2071b" exitCode=0
Sep 11 15:13:18.114031 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:18.114000    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j" event={"ID":"20f43700-b56e-4dff-8fd7-d1d3dc563fbd","Type":"ContainerDied","Data":"89ef67171203b3165a10da01e1216e1d125947c63a7cf4e10cf5714144d2071b"}
Sep 11 15:13:18.114201 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:18.114070    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j" event={"ID":"20f43700-b56e-4dff-8fd7-d1d3dc563fbd","Type":"ContainerDied","Data":"34e748d225c756ddf164c6fac9d7dc34c0143b32121df9fd2b33180880fcc7d9"}
Sep 11 15:13:18.114201 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:18.114018    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j"
Sep 11 15:13:18.121651 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:18.121635    2568 scope.go:117] "RemoveContainer" containerID="7264fbbe73c6a5d5d527217d97e388816b688bcb9879b5d5bfb297ab0f751dae"
Sep 11 15:13:18.122000 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:13:18.121976    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7264fbbe73c6a5d5d527217d97e388816b688bcb9879b5d5bfb297ab0f751dae\": container with ID starting with 7264fbbe73c6a5d5d527217d97e388816b688bcb9879b5d5bfb297ab0f751dae not found: ID does not exist" containerID="7264fbbe73c6a5d5d527217d97e388816b688bcb9879b5d5bfb297ab0f751dae"
Sep 11 15:13:18.122074 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:18.122009    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7264fbbe73c6a5d5d527217d97e388816b688bcb9879b5d5bfb297ab0f751dae"} err="failed to get container status \"7264fbbe73c6a5d5d527217d97e388816b688bcb9879b5d5bfb297ab0f751dae\": rpc error: code = NotFound desc = could not find container \"7264fbbe73c6a5d5d527217d97e388816b688bcb9879b5d5bfb297ab0f751dae\": container with ID starting with 7264fbbe73c6a5d5d527217d97e388816b688bcb9879b5d5bfb297ab0f751dae not found: ID does not exist"
Sep 11 15:13:18.122074 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:18.122024    2568 scope.go:117] "RemoveContainer" containerID="89ef67171203b3165a10da01e1216e1d125947c63a7cf4e10cf5714144d2071b"
Sep 11 15:13:18.130303 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:18.130286    2568 scope.go:117] "RemoveContainer" containerID="89ef67171203b3165a10da01e1216e1d125947c63a7cf4e10cf5714144d2071b"
Sep 11 15:13:18.130563 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:13:18.130545    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ef67171203b3165a10da01e1216e1d125947c63a7cf4e10cf5714144d2071b\": container with ID starting with 89ef67171203b3165a10da01e1216e1d125947c63a7cf4e10cf5714144d2071b not found: ID does not exist" containerID="89ef67171203b3165a10da01e1216e1d125947c63a7cf4e10cf5714144d2071b"
Sep 11 15:13:18.130621 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:18.130571    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ef67171203b3165a10da01e1216e1d125947c63a7cf4e10cf5714144d2071b"} err="failed to get container status \"89ef67171203b3165a10da01e1216e1d125947c63a7cf4e10cf5714144d2071b\": rpc error: code = NotFound desc = could not find container \"89ef67171203b3165a10da01e1216e1d125947c63a7cf4e10cf5714144d2071b\": container with ID starting with 89ef67171203b3165a10da01e1216e1d125947c63a7cf4e10cf5714144d2071b not found: ID does not exist"
Sep 11 15:13:18.138748 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:18.138726    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j"]
Sep 11 15:13:18.145442 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:18.145423    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-177c81-567fbdcbd6-kts6j"]
Sep 11 15:13:18.160011 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:18.159988    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"]
Sep 11 15:13:18.163799 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:18.163779    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-177c81-58cd644b97-rtb8d"]
Sep 11 15:13:18.665550 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:18.665469    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20f43700-b56e-4dff-8fd7-d1d3dc563fbd" path="/var/lib/kubelet/pods/20f43700-b56e-4dff-8fd7-d1d3dc563fbd/volumes"
Sep 11 15:13:18.665922 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:13:18.665898    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="871ed43a-6248-4376-a5e4-a7d43b7db351" path="/var/lib/kubelet/pods/871ed43a-6248-4376-a5e4-a7d43b7db351/volumes"
Sep 11 15:14:06.316744 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:06.316702    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"]
Sep 11 15:14:06.317289 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:06.317014    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l" podUID="46dd338f-8a78-40d4-b447-94b8b63f92b0" containerName="istio-proxy" containerID="cri-o://ece4b0fb2db0fb0edafdaa7ac5d8340a53c4e2d340a54122a1bced574af05ca1" gracePeriod=30
Sep 11 15:14:07.219862 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:07.219801    2568 patch_prober.go:28] interesting pod/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 15:14:07.220042 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:07.219879    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l" podUID="46dd338f-8a78-40d4-b447-94b8b63f92b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 15:14:11.572847 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.572786    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:14:11.742457 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.742371    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-data\") pod \"46dd338f-8a78-40d4-b447-94b8b63f92b0\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") "
Sep 11 15:14:11.742457 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.742405    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-credential-socket\") pod \"46dd338f-8a78-40d4-b447-94b8b63f92b0\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") "
Sep 11 15:14:11.742457 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.742439    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-workload-socket\") pod \"46dd338f-8a78-40d4-b447-94b8b63f92b0\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") "
Sep 11 15:14:11.742775 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.742487    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-envoy\") pod \"46dd338f-8a78-40d4-b447-94b8b63f92b0\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") "
Sep 11 15:14:11.742775 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.742518    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-podinfo\") pod \"46dd338f-8a78-40d4-b447-94b8b63f92b0\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") "
Sep 11 15:14:11.742775 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.742547    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-token\") pod \"46dd338f-8a78-40d4-b447-94b8b63f92b0\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") "
Sep 11 15:14:11.742775 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.742570    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46dd338f-8a78-40d4-b447-94b8b63f92b0-istiod-ca-cert\") pod \"46dd338f-8a78-40d4-b447-94b8b63f92b0\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") "
Sep 11 15:14:11.742775 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.742631    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-data" (OuterVolumeSpecName: "istio-data") pod "46dd338f-8a78-40d4-b447-94b8b63f92b0" (UID: "46dd338f-8a78-40d4-b447-94b8b63f92b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:14:11.742775 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.742691    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "46dd338f-8a78-40d4-b447-94b8b63f92b0" (UID: "46dd338f-8a78-40d4-b447-94b8b63f92b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:14:11.742775 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.742766    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "46dd338f-8a78-40d4-b447-94b8b63f92b0" (UID: "46dd338f-8a78-40d4-b447-94b8b63f92b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:14:11.743131 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.742989    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jg8d\" (UniqueName: \"kubernetes.io/projected/46dd338f-8a78-40d4-b447-94b8b63f92b0-kube-api-access-6jg8d\") pod \"46dd338f-8a78-40d4-b447-94b8b63f92b0\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") "
Sep 11 15:14:11.743131 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.743006    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46dd338f-8a78-40d4-b447-94b8b63f92b0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "46dd338f-8a78-40d4-b447-94b8b63f92b0" (UID: "46dd338f-8a78-40d4-b447-94b8b63f92b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:14:11.743131 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.743030    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-workload-certs\") pod \"46dd338f-8a78-40d4-b447-94b8b63f92b0\" (UID: \"46dd338f-8a78-40d4-b447-94b8b63f92b0\") "
Sep 11 15:14:11.743355 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.743298    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "46dd338f-8a78-40d4-b447-94b8b63f92b0" (UID: "46dd338f-8a78-40d4-b447-94b8b63f92b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:14:11.743489 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.743367    2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46dd338f-8a78-40d4-b447-94b8b63f92b0-istiod-ca-cert\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:11.743489 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.743386    2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-data\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:11.743489 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.743399    2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-credential-socket\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:11.743489 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.743414    2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-workload-socket\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:11.744807 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.744779    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "46dd338f-8a78-40d4-b447-94b8b63f92b0" (UID: "46dd338f-8a78-40d4-b447-94b8b63f92b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 15:14:11.744917 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.744811    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-token" (OuterVolumeSpecName: "istio-token") pod "46dd338f-8a78-40d4-b447-94b8b63f92b0" (UID: "46dd338f-8a78-40d4-b447-94b8b63f92b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:14:11.744917 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.744883    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "46dd338f-8a78-40d4-b447-94b8b63f92b0" (UID: "46dd338f-8a78-40d4-b447-94b8b63f92b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:14:11.744917 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.744884    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46dd338f-8a78-40d4-b447-94b8b63f92b0-kube-api-access-6jg8d" (OuterVolumeSpecName: "kube-api-access-6jg8d") pod "46dd338f-8a78-40d4-b447-94b8b63f92b0" (UID: "46dd338f-8a78-40d4-b447-94b8b63f92b0"). InnerVolumeSpecName "kube-api-access-6jg8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:14:11.844209 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.844163    2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-workload-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:11.844209 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.844203    2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-envoy\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:11.844209 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.844216    2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-podinfo\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:11.844420 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.844227    2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46dd338f-8a78-40d4-b447-94b8b63f92b0-istio-token\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:11.844420 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:11.844240    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jg8d\" (UniqueName: \"kubernetes.io/projected/46dd338f-8a78-40d4-b447-94b8b63f92b0-kube-api-access-6jg8d\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:12.395282 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:12.395246    2568 generic.go:356] "Generic (PLEG): container finished" podID="46dd338f-8a78-40d4-b447-94b8b63f92b0" containerID="ece4b0fb2db0fb0edafdaa7ac5d8340a53c4e2d340a54122a1bced574af05ca1" exitCode=0
Sep 11 15:14:12.395282 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:12.395287    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l" event={"ID":"46dd338f-8a78-40d4-b447-94b8b63f92b0","Type":"ContainerDied","Data":"ece4b0fb2db0fb0edafdaa7ac5d8340a53c4e2d340a54122a1bced574af05ca1"}
Sep 11 15:14:12.395496 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:12.395329    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l" event={"ID":"46dd338f-8a78-40d4-b447-94b8b63f92b0","Type":"ContainerDied","Data":"e7a0cf6bf58e6ecd81ed8a9c31f7f54827fee9a6afcd7d2391dd397e05cfdc67"}
Sep 11 15:14:12.395496 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:12.395353    2568 scope.go:117] "RemoveContainer" containerID="ece4b0fb2db0fb0edafdaa7ac5d8340a53c4e2d340a54122a1bced574af05ca1"
Sep 11 15:14:12.395496 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:12.395289    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"
Sep 11 15:14:12.404593 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:12.404576    2568 scope.go:117] "RemoveContainer" containerID="9a0d624f57698601d2ac5f651e7e5aa1f4296ab1ad69a62175480ca5bfaa9128"
Sep 11 15:14:12.412743 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:12.412726    2568 scope.go:117] "RemoveContainer" containerID="ece4b0fb2db0fb0edafdaa7ac5d8340a53c4e2d340a54122a1bced574af05ca1"
Sep 11 15:14:12.413018 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:14:12.412998    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ece4b0fb2db0fb0edafdaa7ac5d8340a53c4e2d340a54122a1bced574af05ca1\": container with ID starting with ece4b0fb2db0fb0edafdaa7ac5d8340a53c4e2d340a54122a1bced574af05ca1 not found: ID does not exist" containerID="ece4b0fb2db0fb0edafdaa7ac5d8340a53c4e2d340a54122a1bced574af05ca1"
Sep 11 15:14:12.413062 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:12.413027    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece4b0fb2db0fb0edafdaa7ac5d8340a53c4e2d340a54122a1bced574af05ca1"} err="failed to get container status \"ece4b0fb2db0fb0edafdaa7ac5d8340a53c4e2d340a54122a1bced574af05ca1\": rpc error: code = NotFound desc = could not find container \"ece4b0fb2db0fb0edafdaa7ac5d8340a53c4e2d340a54122a1bced574af05ca1\": container with ID starting with ece4b0fb2db0fb0edafdaa7ac5d8340a53c4e2d340a54122a1bced574af05ca1 not found: ID does not exist"
Sep 11 15:14:12.413062 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:12.413044    2568 scope.go:117] "RemoveContainer" containerID="9a0d624f57698601d2ac5f651e7e5aa1f4296ab1ad69a62175480ca5bfaa9128"
Sep 11 15:14:12.413275 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:14:12.413259    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a0d624f57698601d2ac5f651e7e5aa1f4296ab1ad69a62175480ca5bfaa9128\": container with ID starting with 9a0d624f57698601d2ac5f651e7e5aa1f4296ab1ad69a62175480ca5bfaa9128 not found: ID does not exist" containerID="9a0d624f57698601d2ac5f651e7e5aa1f4296ab1ad69a62175480ca5bfaa9128"
Sep 11 15:14:12.413315 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:12.413280    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a0d624f57698601d2ac5f651e7e5aa1f4296ab1ad69a62175480ca5bfaa9128"} err="failed to get container status \"9a0d624f57698601d2ac5f651e7e5aa1f4296ab1ad69a62175480ca5bfaa9128\": rpc error: code = NotFound desc = could not find container \"9a0d624f57698601d2ac5f651e7e5aa1f4296ab1ad69a62175480ca5bfaa9128\": container with ID starting with 9a0d624f57698601d2ac5f651e7e5aa1f4296ab1ad69a62175480ca5bfaa9128 not found: ID does not exist"
Sep 11 15:14:12.417681 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:12.417657    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"]
Sep 11 15:14:12.421662 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:12.421641    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-177c81-openshift-default-69c9bdcf6-6gl8l"]
Sep 11 15:14:12.661601 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:12.661525    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46dd338f-8a78-40d4-b447-94b8b63f92b0" path="/var/lib/kubelet/pods/46dd338f-8a78-40d4-b447-94b8b63f92b0/volumes"
Sep 11 15:14:17.515268 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:17.515230    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t"]
Sep 11 15:14:17.515698 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:17.515476    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t" podUID="3389266a-942d-4a14-9745-7be666ed8b48" containerName="payload-pre-processing" containerID="cri-o://805fe0bcdd22e8f1e50f3a9ccca8a1d90d5d40889927829029f7d8b8f1f7d615" gracePeriod=30
Sep 11 15:14:17.767800 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:17.767740    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t"
Sep 11 15:14:17.890726 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:17.890695    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3389266a-942d-4a14-9745-7be666ed8b48-plugins-config-volume\") pod \"3389266a-942d-4a14-9745-7be666ed8b48\" (UID: \"3389266a-942d-4a14-9745-7be666ed8b48\") "
Sep 11 15:14:17.890927 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:17.890847    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-746v6\" (UniqueName: \"kubernetes.io/projected/3389266a-942d-4a14-9745-7be666ed8b48-kube-api-access-746v6\") pod \"3389266a-942d-4a14-9745-7be666ed8b48\" (UID: \"3389266a-942d-4a14-9745-7be666ed8b48\") "
Sep 11 15:14:17.891092 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:17.891067    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3389266a-942d-4a14-9745-7be666ed8b48-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3389266a-942d-4a14-9745-7be666ed8b48" (UID: "3389266a-942d-4a14-9745-7be666ed8b48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:14:17.892876 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:17.892852    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3389266a-942d-4a14-9745-7be666ed8b48-kube-api-access-746v6" (OuterVolumeSpecName: "kube-api-access-746v6") pod "3389266a-942d-4a14-9745-7be666ed8b48" (UID: "3389266a-942d-4a14-9745-7be666ed8b48"). InnerVolumeSpecName "kube-api-access-746v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:14:17.991639 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:17.991607    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-746v6\" (UniqueName: \"kubernetes.io/projected/3389266a-942d-4a14-9745-7be666ed8b48-kube-api-access-746v6\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:17.991639 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:17.991632    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3389266a-942d-4a14-9745-7be666ed8b48-plugins-config-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:18.429200 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:18.429166    2568 generic.go:356] "Generic (PLEG): container finished" podID="3389266a-942d-4a14-9745-7be666ed8b48" containerID="805fe0bcdd22e8f1e50f3a9ccca8a1d90d5d40889927829029f7d8b8f1f7d615" exitCode=0
Sep 11 15:14:18.429403 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:18.429218    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t"
Sep 11 15:14:18.429403 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:18.429252    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t" event={"ID":"3389266a-942d-4a14-9745-7be666ed8b48","Type":"ContainerDied","Data":"805fe0bcdd22e8f1e50f3a9ccca8a1d90d5d40889927829029f7d8b8f1f7d615"}
Sep 11 15:14:18.429403 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:18.429315    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t" event={"ID":"3389266a-942d-4a14-9745-7be666ed8b48","Type":"ContainerDied","Data":"c55646052e62d3404001703ac2f20012b52e7e9342d6513c93b9ee5551e4734d"}
Sep 11 15:14:18.429403 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:18.429336    2568 scope.go:117] "RemoveContainer" containerID="805fe0bcdd22e8f1e50f3a9ccca8a1d90d5d40889927829029f7d8b8f1f7d615"
Sep 11 15:14:18.438361 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:18.438344    2568 scope.go:117] "RemoveContainer" containerID="805fe0bcdd22e8f1e50f3a9ccca8a1d90d5d40889927829029f7d8b8f1f7d615"
Sep 11 15:14:18.438597 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:14:18.438581    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805fe0bcdd22e8f1e50f3a9ccca8a1d90d5d40889927829029f7d8b8f1f7d615\": container with ID starting with 805fe0bcdd22e8f1e50f3a9ccca8a1d90d5d40889927829029f7d8b8f1f7d615 not found: ID does not exist" containerID="805fe0bcdd22e8f1e50f3a9ccca8a1d90d5d40889927829029f7d8b8f1f7d615"
Sep 11 15:14:18.438646 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:18.438606    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805fe0bcdd22e8f1e50f3a9ccca8a1d90d5d40889927829029f7d8b8f1f7d615"} err="failed to get container status \"805fe0bcdd22e8f1e50f3a9ccca8a1d90d5d40889927829029f7d8b8f1f7d615\": rpc error: code = NotFound desc = could not find container \"805fe0bcdd22e8f1e50f3a9ccca8a1d90d5d40889927829029f7d8b8f1f7d615\": container with ID starting with 805fe0bcdd22e8f1e50f3a9ccca8a1d90d5d40889927829029f7d8b8f1f7d615 not found: ID does not exist"
Sep 11 15:14:18.463630 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:18.463597    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t"]
Sep 11 15:14:18.473601 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:18.473573    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ee29a-5c48cbf4b-dc75t"]
Sep 11 15:14:18.662022 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:18.661985    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3389266a-942d-4a14-9745-7be666ed8b48" path="/var/lib/kubelet/pods/3389266a-942d-4a14-9745-7be666ed8b48/volumes"
Sep 11 15:14:24.171473 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:24.171436    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"]
Sep 11 15:14:24.171907 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:24.171704    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r" podUID="a38d49d7-ae58-46cd-b50c-5c103e52e5fd" containerName="istio-proxy" containerID="cri-o://f3f6408b2afdd3e442c5b4ecabbf603e39c1c3f6649cc97479e75926f3a51769" gracePeriod=30
Sep 11 15:14:29.419012 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.418981    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:14:29.497295 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.497214    2568 generic.go:356] "Generic (PLEG): container finished" podID="a38d49d7-ae58-46cd-b50c-5c103e52e5fd" containerID="f3f6408b2afdd3e442c5b4ecabbf603e39c1c3f6649cc97479e75926f3a51769" exitCode=0
Sep 11 15:14:29.497295 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.497275    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"
Sep 11 15:14:29.497468 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.497297    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r" event={"ID":"a38d49d7-ae58-46cd-b50c-5c103e52e5fd","Type":"ContainerDied","Data":"f3f6408b2afdd3e442c5b4ecabbf603e39c1c3f6649cc97479e75926f3a51769"}
Sep 11 15:14:29.497468 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.497370    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r" event={"ID":"a38d49d7-ae58-46cd-b50c-5c103e52e5fd","Type":"ContainerDied","Data":"eed1b28048c838ff9a6f81ba519468e2b4a9d3b5b556944a3eeac8fd96601947"}
Sep 11 15:14:29.497468 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.497391    2568 scope.go:117] "RemoveContainer" containerID="f3f6408b2afdd3e442c5b4ecabbf603e39c1c3f6649cc97479e75926f3a51769"
Sep 11 15:14:29.502833 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.502784    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-token\") pod \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") "
Sep 11 15:14:29.502959 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.502898    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-envoy\") pod \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") "
Sep 11 15:14:29.502959 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.502928    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-credential-socket\") pod \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") "
Sep 11 15:14:29.502959 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.502956    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5x7ll\" (UniqueName: \"kubernetes.io/projected/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-kube-api-access-5x7ll\") pod \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") "
Sep 11 15:14:29.503126 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.502981    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-podinfo\") pod \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") "
Sep 11 15:14:29.503126 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.503043    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-data\") pod \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") "
Sep 11 15:14:29.503126 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.503099    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-workload-socket\") pod \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") "
Sep 11 15:14:29.503280 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.503141    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istiod-ca-cert\") pod \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") "
Sep 11 15:14:29.503280 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.503168    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-workload-certs\") pod \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\" (UID: \"a38d49d7-ae58-46cd-b50c-5c103e52e5fd\") "
Sep 11 15:14:29.503280 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.503226    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a38d49d7-ae58-46cd-b50c-5c103e52e5fd" (UID: "a38d49d7-ae58-46cd-b50c-5c103e52e5fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:14:29.503464 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.503444    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a38d49d7-ae58-46cd-b50c-5c103e52e5fd" (UID: "a38d49d7-ae58-46cd-b50c-5c103e52e5fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:14:29.503582 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.503525    2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-credential-socket\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:29.503653 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.503552    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a38d49d7-ae58-46cd-b50c-5c103e52e5fd" (UID: "a38d49d7-ae58-46cd-b50c-5c103e52e5fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:14:29.503712 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.503650    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-data" (OuterVolumeSpecName: "istio-data") pod "a38d49d7-ae58-46cd-b50c-5c103e52e5fd" (UID: "a38d49d7-ae58-46cd-b50c-5c103e52e5fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:14:29.503843 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.503804    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a38d49d7-ae58-46cd-b50c-5c103e52e5fd" (UID: "a38d49d7-ae58-46cd-b50c-5c103e52e5fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:14:29.505354 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.505332    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-kube-api-access-5x7ll" (OuterVolumeSpecName: "kube-api-access-5x7ll") pod "a38d49d7-ae58-46cd-b50c-5c103e52e5fd" (UID: "a38d49d7-ae58-46cd-b50c-5c103e52e5fd"). InnerVolumeSpecName "kube-api-access-5x7ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:14:29.505473 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.505364    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-token" (OuterVolumeSpecName: "istio-token") pod "a38d49d7-ae58-46cd-b50c-5c103e52e5fd" (UID: "a38d49d7-ae58-46cd-b50c-5c103e52e5fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:14:29.505473 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.505439    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a38d49d7-ae58-46cd-b50c-5c103e52e5fd" (UID: "a38d49d7-ae58-46cd-b50c-5c103e52e5fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 15:14:29.505575 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.505505    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a38d49d7-ae58-46cd-b50c-5c103e52e5fd" (UID: "a38d49d7-ae58-46cd-b50c-5c103e52e5fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 15:14:29.506629 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.506609    2568 scope.go:117] "RemoveContainer" containerID="b64ba9b82346b63708090e9446722f5647c9f2226c08dd1c7e12857ef81aeb00"
Sep 11 15:14:29.523603 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.523577    2568 scope.go:117] "RemoveContainer" containerID="f3f6408b2afdd3e442c5b4ecabbf603e39c1c3f6649cc97479e75926f3a51769"
Sep 11 15:14:29.523901 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:14:29.523882    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f6408b2afdd3e442c5b4ecabbf603e39c1c3f6649cc97479e75926f3a51769\": container with ID starting with f3f6408b2afdd3e442c5b4ecabbf603e39c1c3f6649cc97479e75926f3a51769 not found: ID does not exist" containerID="f3f6408b2afdd3e442c5b4ecabbf603e39c1c3f6649cc97479e75926f3a51769"
Sep 11 15:14:29.523956 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.523909    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f6408b2afdd3e442c5b4ecabbf603e39c1c3f6649cc97479e75926f3a51769"} err="failed to get container status \"f3f6408b2afdd3e442c5b4ecabbf603e39c1c3f6649cc97479e75926f3a51769\": rpc error: code = NotFound desc = could not find container \"f3f6408b2afdd3e442c5b4ecabbf603e39c1c3f6649cc97479e75926f3a51769\": container with ID starting with f3f6408b2afdd3e442c5b4ecabbf603e39c1c3f6649cc97479e75926f3a51769 not found: ID does not exist"
Sep 11 15:14:29.523956 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.523925    2568 scope.go:117] "RemoveContainer" containerID="b64ba9b82346b63708090e9446722f5647c9f2226c08dd1c7e12857ef81aeb00"
Sep 11 15:14:29.524165 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:14:29.524148    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b64ba9b82346b63708090e9446722f5647c9f2226c08dd1c7e12857ef81aeb00\": container with ID starting with b64ba9b82346b63708090e9446722f5647c9f2226c08dd1c7e12857ef81aeb00 not found: ID does not exist" containerID="b64ba9b82346b63708090e9446722f5647c9f2226c08dd1c7e12857ef81aeb00"
Sep 11 15:14:29.524208 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.524171    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b64ba9b82346b63708090e9446722f5647c9f2226c08dd1c7e12857ef81aeb00"} err="failed to get container status \"b64ba9b82346b63708090e9446722f5647c9f2226c08dd1c7e12857ef81aeb00\": rpc error: code = NotFound desc = could not find container \"b64ba9b82346b63708090e9446722f5647c9f2226c08dd1c7e12857ef81aeb00\": container with ID starting with b64ba9b82346b63708090e9446722f5647c9f2226c08dd1c7e12857ef81aeb00 not found: ID does not exist"
Sep 11 15:14:29.604335 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.604303    2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istiod-ca-cert\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:29.604335 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.604328    2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-workload-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:29.604335 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.604340    2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-token\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:29.604524 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.604351    2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-envoy\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:29.604524 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.604363    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x7ll\" (UniqueName: \"kubernetes.io/projected/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-kube-api-access-5x7ll\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:29.604524 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.604374    2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-podinfo\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:29.604524 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.604386    2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-istio-data\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:29.604524 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.604397    2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38d49d7-ae58-46cd-b50c-5c103e52e5fd-workload-socket\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:29.820294 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.820259    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"]
Sep 11 15:14:29.823633 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:29.823607    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4ee29a-openshift-default-7c9dfbccd4-n7b7r"]
Sep 11 15:14:30.661308 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:30.661278    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a38d49d7-ae58-46cd-b50c-5c103e52e5fd" path="/var/lib/kubelet/pods/a38d49d7-ae58-46cd-b50c-5c103e52e5fd/volumes"
Sep 11 15:14:35.050320 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:35.050284    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2"]
Sep 11 15:14:35.050745 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:35.050494    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2" podUID="40f2dbf4-4caf-4623-adcc-9079eb94ed7a" containerName="payload-processing" containerID="cri-o://3961c68ccc92499ce3bca21fcfa8ad5c4a44e9286b80b880ed87abe9eb0c9014" gracePeriod=30
Sep 11 15:14:35.316194 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:35.316125    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2"
Sep 11 15:14:35.459551 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:35.459516    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f2dbf4-4caf-4623-adcc-9079eb94ed7a-plugins-config-volume\") pod \"40f2dbf4-4caf-4623-adcc-9079eb94ed7a\" (UID: \"40f2dbf4-4caf-4623-adcc-9079eb94ed7a\") "
Sep 11 15:14:35.459738 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:35.459605    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cszg\" (UniqueName: \"kubernetes.io/projected/40f2dbf4-4caf-4623-adcc-9079eb94ed7a-kube-api-access-9cszg\") pod \"40f2dbf4-4caf-4623-adcc-9079eb94ed7a\" (UID: \"40f2dbf4-4caf-4623-adcc-9079eb94ed7a\") "
Sep 11 15:14:35.462328 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:35.462107    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f2dbf4-4caf-4623-adcc-9079eb94ed7a-kube-api-access-9cszg" (OuterVolumeSpecName: "kube-api-access-9cszg") pod "40f2dbf4-4caf-4623-adcc-9079eb94ed7a" (UID: "40f2dbf4-4caf-4623-adcc-9079eb94ed7a"). InnerVolumeSpecName "kube-api-access-9cszg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:14:35.462644 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:35.462613    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f2dbf4-4caf-4623-adcc-9079eb94ed7a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "40f2dbf4-4caf-4623-adcc-9079eb94ed7a" (UID: "40f2dbf4-4caf-4623-adcc-9079eb94ed7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:14:35.531431 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:35.531402    2568 generic.go:356] "Generic (PLEG): container finished" podID="40f2dbf4-4caf-4623-adcc-9079eb94ed7a" containerID="3961c68ccc92499ce3bca21fcfa8ad5c4a44e9286b80b880ed87abe9eb0c9014" exitCode=0
Sep 11 15:14:35.531560 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:35.531437    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2" event={"ID":"40f2dbf4-4caf-4623-adcc-9079eb94ed7a","Type":"ContainerDied","Data":"3961c68ccc92499ce3bca21fcfa8ad5c4a44e9286b80b880ed87abe9eb0c9014"}
Sep 11 15:14:35.531560 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:35.531476    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2"
Sep 11 15:14:35.531560 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:35.531500    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2" event={"ID":"40f2dbf4-4caf-4623-adcc-9079eb94ed7a","Type":"ContainerDied","Data":"d23b4cebc986d5a70dcd5d4fcb6924d3c619f12a0ed248194b73f26d94e1566e"}
Sep 11 15:14:35.531560 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:35.531521    2568 scope.go:117] "RemoveContainer" containerID="3961c68ccc92499ce3bca21fcfa8ad5c4a44e9286b80b880ed87abe9eb0c9014"
Sep 11 15:14:35.540158 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:35.540141    2568 scope.go:117] "RemoveContainer" containerID="3961c68ccc92499ce3bca21fcfa8ad5c4a44e9286b80b880ed87abe9eb0c9014"
Sep 11 15:14:35.540422 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:14:35.540402    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3961c68ccc92499ce3bca21fcfa8ad5c4a44e9286b80b880ed87abe9eb0c9014\": container with ID starting with 3961c68ccc92499ce3bca21fcfa8ad5c4a44e9286b80b880ed87abe9eb0c9014 not found: ID does not exist" containerID="3961c68ccc92499ce3bca21fcfa8ad5c4a44e9286b80b880ed87abe9eb0c9014"
Sep 11 15:14:35.540469 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:35.540429    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3961c68ccc92499ce3bca21fcfa8ad5c4a44e9286b80b880ed87abe9eb0c9014"} err="failed to get container status \"3961c68ccc92499ce3bca21fcfa8ad5c4a44e9286b80b880ed87abe9eb0c9014\": rpc error: code = NotFound desc = could not find container \"3961c68ccc92499ce3bca21fcfa8ad5c4a44e9286b80b880ed87abe9eb0c9014\": container with ID starting with 3961c68ccc92499ce3bca21fcfa8ad5c4a44e9286b80b880ed87abe9eb0c9014 not found: ID does not exist"
Sep 11 15:14:35.555496 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:35.555473    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2"]
Sep 11 15:14:35.560245 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:35.560220    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-48898f-797669c7bf-ql6l2"]
Sep 11 15:14:35.560549 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:35.560531    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f2dbf4-4caf-4623-adcc-9079eb94ed7a-plugins-config-volume\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:35.560591 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:35.560553    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cszg\" (UniqueName: \"kubernetes.io/projected/40f2dbf4-4caf-4623-adcc-9079eb94ed7a-kube-api-access-9cszg\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:36.662522 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:36.662485    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40f2dbf4-4caf-4623-adcc-9079eb94ed7a" path="/var/lib/kubelet/pods/40f2dbf4-4caf-4623-adcc-9079eb94ed7a/volumes"
Sep 11 15:14:52.969942 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:52.969666    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"]
Sep 11 15:14:52.970591 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:52.970536    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls" podUID="baaaba03-d432-4125-9978-7d63d9f7e5ca" containerName="maas-api" containerID="cri-o://c8fa8ebd9e26d4f9e6e5f4a4a0edeba7df74e763785f4200f2d3aead9614d4bd" gracePeriod=30
Sep 11 15:14:53.224655 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.224589    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"
Sep 11 15:14:53.315769 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.315733    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baaaba03-d432-4125-9978-7d63d9f7e5ca-openshift-service-ca\") pod \"baaaba03-d432-4125-9978-7d63d9f7e5ca\" (UID: \"baaaba03-d432-4125-9978-7d63d9f7e5ca\") "
Sep 11 15:14:53.315968 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.315801    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbwwj\" (UniqueName: \"kubernetes.io/projected/baaaba03-d432-4125-9978-7d63d9f7e5ca-kube-api-access-sbwwj\") pod \"baaaba03-d432-4125-9978-7d63d9f7e5ca\" (UID: \"baaaba03-d432-4125-9978-7d63d9f7e5ca\") "
Sep 11 15:14:53.315968 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.315878    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baaaba03-d432-4125-9978-7d63d9f7e5ca-metrics-certs\") pod \"baaaba03-d432-4125-9978-7d63d9f7e5ca\" (UID: \"baaaba03-d432-4125-9978-7d63d9f7e5ca\") "
Sep 11 15:14:53.315968 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.315926    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baaaba03-d432-4125-9978-7d63d9f7e5ca-maas-api-tls\") pod \"baaaba03-d432-4125-9978-7d63d9f7e5ca\" (UID: \"baaaba03-d432-4125-9978-7d63d9f7e5ca\") "
Sep 11 15:14:53.316344 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.316201    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baaaba03-d432-4125-9978-7d63d9f7e5ca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "baaaba03-d432-4125-9978-7d63d9f7e5ca" (UID: "baaaba03-d432-4125-9978-7d63d9f7e5ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:14:53.316344 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.316316    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baaaba03-d432-4125-9978-7d63d9f7e5ca-openshift-service-ca\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:53.318738 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.318712    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baaaba03-d432-4125-9978-7d63d9f7e5ca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "baaaba03-d432-4125-9978-7d63d9f7e5ca" (UID: "baaaba03-d432-4125-9978-7d63d9f7e5ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:14:53.318869 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.318856    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baaaba03-d432-4125-9978-7d63d9f7e5ca-kube-api-access-sbwwj" (OuterVolumeSpecName: "kube-api-access-sbwwj") pod "baaaba03-d432-4125-9978-7d63d9f7e5ca" (UID: "baaaba03-d432-4125-9978-7d63d9f7e5ca"). InnerVolumeSpecName "kube-api-access-sbwwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:14:53.318950 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.318917    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baaaba03-d432-4125-9978-7d63d9f7e5ca-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "baaaba03-d432-4125-9978-7d63d9f7e5ca" (UID: "baaaba03-d432-4125-9978-7d63d9f7e5ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:14:53.417010 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.416971    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbwwj\" (UniqueName: \"kubernetes.io/projected/baaaba03-d432-4125-9978-7d63d9f7e5ca-kube-api-access-sbwwj\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:53.417010 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.417010    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baaaba03-d432-4125-9978-7d63d9f7e5ca-metrics-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:53.417206 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.417025    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baaaba03-d432-4125-9978-7d63d9f7e5ca-maas-api-tls\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:14:53.619449 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.619414    2568 generic.go:356] "Generic (PLEG): container finished" podID="baaaba03-d432-4125-9978-7d63d9f7e5ca" containerID="c8fa8ebd9e26d4f9e6e5f4a4a0edeba7df74e763785f4200f2d3aead9614d4bd" exitCode=0
Sep 11 15:14:53.619449 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.619450    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"
Sep 11 15:14:53.619647 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.619488    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls" event={"ID":"baaaba03-d432-4125-9978-7d63d9f7e5ca","Type":"ContainerDied","Data":"c8fa8ebd9e26d4f9e6e5f4a4a0edeba7df74e763785f4200f2d3aead9614d4bd"}
Sep 11 15:14:53.619647 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.619536    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls" event={"ID":"baaaba03-d432-4125-9978-7d63d9f7e5ca","Type":"ContainerDied","Data":"2d94f83c53cb5b126d348b5782b4ceaf143cff1ca0ead65cd1396e547d07d08e"}
Sep 11 15:14:53.619647 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.619561    2568 scope.go:117] "RemoveContainer" containerID="c8fa8ebd9e26d4f9e6e5f4a4a0edeba7df74e763785f4200f2d3aead9614d4bd"
Sep 11 15:14:53.629332 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.629315    2568 scope.go:117] "RemoveContainer" containerID="c8fa8ebd9e26d4f9e6e5f4a4a0edeba7df74e763785f4200f2d3aead9614d4bd"
Sep 11 15:14:53.629609 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:14:53.629588    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8fa8ebd9e26d4f9e6e5f4a4a0edeba7df74e763785f4200f2d3aead9614d4bd\": container with ID starting with c8fa8ebd9e26d4f9e6e5f4a4a0edeba7df74e763785f4200f2d3aead9614d4bd not found: ID does not exist" containerID="c8fa8ebd9e26d4f9e6e5f4a4a0edeba7df74e763785f4200f2d3aead9614d4bd"
Sep 11 15:14:53.629678 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.629618    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8fa8ebd9e26d4f9e6e5f4a4a0edeba7df74e763785f4200f2d3aead9614d4bd"} err="failed to get container status \"c8fa8ebd9e26d4f9e6e5f4a4a0edeba7df74e763785f4200f2d3aead9614d4bd\": rpc error: code = NotFound desc = could not find container \"c8fa8ebd9e26d4f9e6e5f4a4a0edeba7df74e763785f4200f2d3aead9614d4bd\": container with ID starting with c8fa8ebd9e26d4f9e6e5f4a4a0edeba7df74e763785f4200f2d3aead9614d4bd not found: ID does not exist"
Sep 11 15:14:53.641069 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.641045    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"]
Sep 11 15:14:53.648074 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:53.648045    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-928943-6c759c58f9-29sls"]
Sep 11 15:14:54.662486 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:14:54.662451    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baaaba03-d432-4125-9978-7d63d9f7e5ca" path="/var/lib/kubelet/pods/baaaba03-d432-4125-9978-7d63d9f7e5ca/volumes"
Sep 11 15:15:02.750307 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:02.750266    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bdb5c7ff-jz27x"]
Sep 11 15:15:02.750842 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:02.750494    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5bdb5c7ff-jz27x" podUID="6a191532-c062-4aed-ab16-771a4bb32f16" containerName="manager" containerID="cri-o://9d692337f53edaf4e429076d58ff9d96824bf04dad304b45912de520895aad52" gracePeriod=30
Sep 11 15:15:02.996564 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:02.996533    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bdb5c7ff-jz27x"
Sep 11 15:15:03.005221 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:03.005163    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a191532-c062-4aed-ab16-771a4bb32f16-metrics-certs\") pod \"6a191532-c062-4aed-ab16-771a4bb32f16\" (UID: \"6a191532-c062-4aed-ab16-771a4bb32f16\") "
Sep 11 15:15:03.005318 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:03.005219    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tmtc\" (UniqueName: \"kubernetes.io/projected/6a191532-c062-4aed-ab16-771a4bb32f16-kube-api-access-8tmtc\") pod \"6a191532-c062-4aed-ab16-771a4bb32f16\" (UID: \"6a191532-c062-4aed-ab16-771a4bb32f16\") "
Sep 11 15:15:03.005318 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:03.005294    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a191532-c062-4aed-ab16-771a4bb32f16-webhook-cert\") pod \"6a191532-c062-4aed-ab16-771a4bb32f16\" (UID: \"6a191532-c062-4aed-ab16-771a4bb32f16\") "
Sep 11 15:15:03.007290 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:03.007260    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a191532-c062-4aed-ab16-771a4bb32f16-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6a191532-c062-4aed-ab16-771a4bb32f16" (UID: "6a191532-c062-4aed-ab16-771a4bb32f16"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:15:03.007397 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:03.007289    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a191532-c062-4aed-ab16-771a4bb32f16-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6a191532-c062-4aed-ab16-771a4bb32f16" (UID: "6a191532-c062-4aed-ab16-771a4bb32f16"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:15:03.007397 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:03.007339    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a191532-c062-4aed-ab16-771a4bb32f16-kube-api-access-8tmtc" (OuterVolumeSpecName: "kube-api-access-8tmtc") pod "6a191532-c062-4aed-ab16-771a4bb32f16" (UID: "6a191532-c062-4aed-ab16-771a4bb32f16"). InnerVolumeSpecName "kube-api-access-8tmtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:15:03.106626 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:03.106596    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tmtc\" (UniqueName: \"kubernetes.io/projected/6a191532-c062-4aed-ab16-771a4bb32f16-kube-api-access-8tmtc\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:15:03.106626 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:03.106624    2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a191532-c062-4aed-ab16-771a4bb32f16-webhook-cert\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:15:03.106626 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:03.106633    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a191532-c062-4aed-ab16-771a4bb32f16-metrics-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:15:03.671144 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:03.671108    2568 generic.go:356] "Generic (PLEG): container finished" podID="6a191532-c062-4aed-ab16-771a4bb32f16" containerID="9d692337f53edaf4e429076d58ff9d96824bf04dad304b45912de520895aad52" exitCode=0
Sep 11 15:15:03.671144 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:03.671143    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bdb5c7ff-jz27x"
Sep 11 15:15:03.671408 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:03.671196    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bdb5c7ff-jz27x" event={"ID":"6a191532-c062-4aed-ab16-771a4bb32f16","Type":"ContainerDied","Data":"9d692337f53edaf4e429076d58ff9d96824bf04dad304b45912de520895aad52"}
Sep 11 15:15:03.671408 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:03.671239    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bdb5c7ff-jz27x" event={"ID":"6a191532-c062-4aed-ab16-771a4bb32f16","Type":"ContainerDied","Data":"a69521207d514ed05d5c2a992f3f05483d87f56e66ec8514f16425e8582304f9"}
Sep 11 15:15:03.671408 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:03.671262    2568 scope.go:117] "RemoveContainer" containerID="9d692337f53edaf4e429076d58ff9d96824bf04dad304b45912de520895aad52"
Sep 11 15:15:03.681191 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:03.681172    2568 scope.go:117] "RemoveContainer" containerID="9d692337f53edaf4e429076d58ff9d96824bf04dad304b45912de520895aad52"
Sep 11 15:15:03.681476 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:15:03.681459    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d692337f53edaf4e429076d58ff9d96824bf04dad304b45912de520895aad52\": container with ID starting with 9d692337f53edaf4e429076d58ff9d96824bf04dad304b45912de520895aad52 not found: ID does not exist" containerID="9d692337f53edaf4e429076d58ff9d96824bf04dad304b45912de520895aad52"
Sep 11 15:15:03.681520 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:03.681484    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d692337f53edaf4e429076d58ff9d96824bf04dad304b45912de520895aad52"} err="failed to get container status \"9d692337f53edaf4e429076d58ff9d96824bf04dad304b45912de520895aad52\": rpc error: code = NotFound desc = could not find container \"9d692337f53edaf4e429076d58ff9d96824bf04dad304b45912de520895aad52\": container with ID starting with 9d692337f53edaf4e429076d58ff9d96824bf04dad304b45912de520895aad52 not found: ID does not exist"
Sep 11 15:15:03.693068 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:03.693039    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bdb5c7ff-jz27x"]
Sep 11 15:15:03.696361 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:03.696340    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5bdb5c7ff-jz27x"]
Sep 11 15:15:04.661205 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:15:04.661172    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a191532-c062-4aed-ab16-771a4bb32f16" path="/var/lib/kubelet/pods/6a191532-c062-4aed-ab16-771a4bb32f16/volumes"
Sep 11 15:16:04.500964 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.500928    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bdb5c7ff-4qcw9"]
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501306    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40f2dbf4-4caf-4623-adcc-9079eb94ed7a" containerName="payload-processing"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501317    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="40f2dbf4-4caf-4623-adcc-9079eb94ed7a" containerName="payload-processing"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501332    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a38d49d7-ae58-46cd-b50c-5c103e52e5fd" containerName="istio-proxy"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501338    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a38d49d7-ae58-46cd-b50c-5c103e52e5fd" containerName="istio-proxy"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501350    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af900bff-c3fd-4d8f-968b-962c71d078bd" containerName="payload-processing"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501355    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="af900bff-c3fd-4d8f-968b-962c71d078bd" containerName="payload-processing"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501365    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a191532-c062-4aed-ab16-771a4bb32f16" containerName="manager"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501370    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a191532-c062-4aed-ab16-771a4bb32f16" containerName="manager"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501377    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3389266a-942d-4a14-9745-7be666ed8b48" containerName="payload-pre-processing"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501382    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3389266a-942d-4a14-9745-7be666ed8b48" containerName="payload-pre-processing"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501388    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46dd338f-8a78-40d4-b447-94b8b63f92b0" containerName="istio-proxy"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501393    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="46dd338f-8a78-40d4-b447-94b8b63f92b0" containerName="istio-proxy"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501400    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46dd338f-8a78-40d4-b447-94b8b63f92b0" containerName="istio-proxy"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501405    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="46dd338f-8a78-40d4-b447-94b8b63f92b0" containerName="istio-proxy"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501412    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0f069f-98b9-45d8-9058-1e582f39e219" containerName="maas-api"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501417    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0f069f-98b9-45d8-9058-1e582f39e219" containerName="maas-api"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501427    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a38d49d7-ae58-46cd-b50c-5c103e52e5fd" containerName="istio-proxy"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501431    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a38d49d7-ae58-46cd-b50c-5c103e52e5fd" containerName="istio-proxy"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501437    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baaaba03-d432-4125-9978-7d63d9f7e5ca" containerName="maas-api"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501443    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="baaaba03-d432-4125-9978-7d63d9f7e5ca" containerName="maas-api"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501451    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20f43700-b56e-4dff-8fd7-d1d3dc563fbd" containerName="payload-pre-processing"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501456    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="20f43700-b56e-4dff-8fd7-d1d3dc563fbd" containerName="payload-pre-processing"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501461    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="871ed43a-6248-4376-a5e4-a7d43b7db351" containerName="maas-api"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501474    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="871ed43a-6248-4376-a5e4-a7d43b7db351" containerName="maas-api"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501526    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a38d49d7-ae58-46cd-b50c-5c103e52e5fd" containerName="istio-proxy"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501534    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="46dd338f-8a78-40d4-b447-94b8b63f92b0" containerName="istio-proxy"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501540    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a0f069f-98b9-45d8-9058-1e582f39e219" containerName="maas-api"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501545    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a191532-c062-4aed-ab16-771a4bb32f16" containerName="manager"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501550    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="46dd338f-8a78-40d4-b447-94b8b63f92b0" containerName="istio-proxy"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501556    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="20f43700-b56e-4dff-8fd7-d1d3dc563fbd" containerName="payload-pre-processing"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501560    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="871ed43a-6248-4376-a5e4-a7d43b7db351" containerName="maas-api"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501566    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3389266a-942d-4a14-9745-7be666ed8b48" containerName="payload-pre-processing"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501573    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="baaaba03-d432-4125-9978-7d63d9f7e5ca" containerName="maas-api"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501580    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="af900bff-c3fd-4d8f-968b-962c71d078bd" containerName="payload-processing"
Sep 11 15:16:04.503311 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.501585    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="40f2dbf4-4caf-4623-adcc-9079eb94ed7a" containerName="payload-processing"
Sep 11 15:16:04.504447 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.504430    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9"
Sep 11 15:16:04.508129 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.508099    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 15:16:04.508129 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.508117    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 15:16:04.508260 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.508117    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 15:16:04.508260 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.508112    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sd75p\""
Sep 11 15:16:04.513705 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.513682    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bdb5c7ff-4qcw9"]
Sep 11 15:16:04.529491 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.529466    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80930310-9725-4223-b1ad-8f486bf0a119-metrics-certs\") pod \"maas-controller-5bdb5c7ff-4qcw9\" (UID: \"80930310-9725-4223-b1ad-8f486bf0a119\") " pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9"
Sep 11 15:16:04.529610 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.529498    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9v72\" (UniqueName: \"kubernetes.io/projected/80930310-9725-4223-b1ad-8f486bf0a119-kube-api-access-z9v72\") pod \"maas-controller-5bdb5c7ff-4qcw9\" (UID: \"80930310-9725-4223-b1ad-8f486bf0a119\") " pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9"
Sep 11 15:16:04.529610 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.529519    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80930310-9725-4223-b1ad-8f486bf0a119-webhook-cert\") pod \"maas-controller-5bdb5c7ff-4qcw9\" (UID: \"80930310-9725-4223-b1ad-8f486bf0a119\") " pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9"
Sep 11 15:16:04.630960 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.630925    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80930310-9725-4223-b1ad-8f486bf0a119-metrics-certs\") pod \"maas-controller-5bdb5c7ff-4qcw9\" (UID: \"80930310-9725-4223-b1ad-8f486bf0a119\") " pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9"
Sep 11 15:16:04.630960 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.630961    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9v72\" (UniqueName: \"kubernetes.io/projected/80930310-9725-4223-b1ad-8f486bf0a119-kube-api-access-z9v72\") pod \"maas-controller-5bdb5c7ff-4qcw9\" (UID: \"80930310-9725-4223-b1ad-8f486bf0a119\") " pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9"
Sep 11 15:16:04.631168 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.630980    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80930310-9725-4223-b1ad-8f486bf0a119-webhook-cert\") pod \"maas-controller-5bdb5c7ff-4qcw9\" (UID: \"80930310-9725-4223-b1ad-8f486bf0a119\") " pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9"
Sep 11 15:16:04.633534 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.633503    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80930310-9725-4223-b1ad-8f486bf0a119-webhook-cert\") pod \"maas-controller-5bdb5c7ff-4qcw9\" (UID: \"80930310-9725-4223-b1ad-8f486bf0a119\") " pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9"
Sep 11 15:16:04.633655 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.633562    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80930310-9725-4223-b1ad-8f486bf0a119-metrics-certs\") pod \"maas-controller-5bdb5c7ff-4qcw9\" (UID: \"80930310-9725-4223-b1ad-8f486bf0a119\") " pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9"
Sep 11 15:16:04.638429 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.638397    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9v72\" (UniqueName: \"kubernetes.io/projected/80930310-9725-4223-b1ad-8f486bf0a119-kube-api-access-z9v72\") pod \"maas-controller-5bdb5c7ff-4qcw9\" (UID: \"80930310-9725-4223-b1ad-8f486bf0a119\") " pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9"
Sep 11 15:16:04.816006 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:04.815980    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9"
Sep 11 15:16:05.088518 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:05.088494    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bdb5c7ff-4qcw9"]
Sep 11 15:16:05.090762 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:16:05.090735    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80930310_9725_4223_b1ad_8f486bf0a119.slice/crio-e7fb249ba0c5283dd29c7d34af17f55ead01cb6507d6b6e2ca8aefb547d2e6ab WatchSource:0}: Error finding container e7fb249ba0c5283dd29c7d34af17f55ead01cb6507d6b6e2ca8aefb547d2e6ab: Status 404 returned error can't find the container with id e7fb249ba0c5283dd29c7d34af17f55ead01cb6507d6b6e2ca8aefb547d2e6ab
Sep 11 15:16:05.092152 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:05.092133    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 15:16:05.957973 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:05.957938    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9" event={"ID":"80930310-9725-4223-b1ad-8f486bf0a119","Type":"ContainerStarted","Data":"d129728156de9a4623be08be61bd78b2bd7f8b8da4819982e55c8d5e73c60bab"}
Sep 11 15:16:05.958461 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:05.957986    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9" event={"ID":"80930310-9725-4223-b1ad-8f486bf0a119","Type":"ContainerStarted","Data":"e7fb249ba0c5283dd29c7d34af17f55ead01cb6507d6b6e2ca8aefb547d2e6ab"}
Sep 11 15:16:05.958461 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:05.958071    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9"
Sep 11 15:16:05.975752 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:05.975702    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9" podStartSLOduration=1.546063079 podStartE2EDuration="1.975688171s" totalImagesPullingTime="429.625092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:16:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:16:05.97475269 +0000 UTC m=+2145.763016988" watchObservedRunningTime="2026-09-11 15:16:05.975688171 +0000 UTC m=+2145.763952484"
Sep 11 15:16:16.964770 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:16.964691    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9"
Sep 11 15:16:26.348513 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:26.348466    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bdb5c7ff-4qcw9"]
Sep 11 15:16:26.349015 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:26.348686    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9" podUID="80930310-9725-4223-b1ad-8f486bf0a119" containerName="manager" containerID="cri-o://d129728156de9a4623be08be61bd78b2bd7f8b8da4819982e55c8d5e73c60bab" gracePeriod=30
Sep 11 15:16:26.589445 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:26.589418    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9"
Sep 11 15:16:26.724062 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:26.723979    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80930310-9725-4223-b1ad-8f486bf0a119-webhook-cert\") pod \"80930310-9725-4223-b1ad-8f486bf0a119\" (UID: \"80930310-9725-4223-b1ad-8f486bf0a119\") "
Sep 11 15:16:26.724223 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:26.724066    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9v72\" (UniqueName: \"kubernetes.io/projected/80930310-9725-4223-b1ad-8f486bf0a119-kube-api-access-z9v72\") pod \"80930310-9725-4223-b1ad-8f486bf0a119\" (UID: \"80930310-9725-4223-b1ad-8f486bf0a119\") "
Sep 11 15:16:26.724223 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:26.724107    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80930310-9725-4223-b1ad-8f486bf0a119-metrics-certs\") pod \"80930310-9725-4223-b1ad-8f486bf0a119\" (UID: \"80930310-9725-4223-b1ad-8f486bf0a119\") "
Sep 11 15:16:26.726126 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:26.726091    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80930310-9725-4223-b1ad-8f486bf0a119-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "80930310-9725-4223-b1ad-8f486bf0a119" (UID: "80930310-9725-4223-b1ad-8f486bf0a119"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:16:26.726239 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:26.726100    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80930310-9725-4223-b1ad-8f486bf0a119-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "80930310-9725-4223-b1ad-8f486bf0a119" (UID: "80930310-9725-4223-b1ad-8f486bf0a119"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:16:26.726239 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:26.726129    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80930310-9725-4223-b1ad-8f486bf0a119-kube-api-access-z9v72" (OuterVolumeSpecName: "kube-api-access-z9v72") pod "80930310-9725-4223-b1ad-8f486bf0a119" (UID: "80930310-9725-4223-b1ad-8f486bf0a119"). InnerVolumeSpecName "kube-api-access-z9v72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:16:26.825629 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:26.825587    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80930310-9725-4223-b1ad-8f486bf0a119-metrics-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:16:26.825629 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:26.825627    2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80930310-9725-4223-b1ad-8f486bf0a119-webhook-cert\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:16:26.825629 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:26.825636    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9v72\" (UniqueName: \"kubernetes.io/projected/80930310-9725-4223-b1ad-8f486bf0a119-kube-api-access-z9v72\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:16:27.055230 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:27.055192    2568 generic.go:356] "Generic (PLEG): container finished" podID="80930310-9725-4223-b1ad-8f486bf0a119" containerID="d129728156de9a4623be08be61bd78b2bd7f8b8da4819982e55c8d5e73c60bab" exitCode=0
Sep 11 15:16:27.055413 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:27.055239    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9"
Sep 11 15:16:27.055413 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:27.055259    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9" event={"ID":"80930310-9725-4223-b1ad-8f486bf0a119","Type":"ContainerDied","Data":"d129728156de9a4623be08be61bd78b2bd7f8b8da4819982e55c8d5e73c60bab"}
Sep 11 15:16:27.055413 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:27.055310    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bdb5c7ff-4qcw9" event={"ID":"80930310-9725-4223-b1ad-8f486bf0a119","Type":"ContainerDied","Data":"e7fb249ba0c5283dd29c7d34af17f55ead01cb6507d6b6e2ca8aefb547d2e6ab"}
Sep 11 15:16:27.055413 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:27.055340    2568 scope.go:117] "RemoveContainer" containerID="d129728156de9a4623be08be61bd78b2bd7f8b8da4819982e55c8d5e73c60bab"
Sep 11 15:16:27.064116 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:27.064098    2568 scope.go:117] "RemoveContainer" containerID="d129728156de9a4623be08be61bd78b2bd7f8b8da4819982e55c8d5e73c60bab"
Sep 11 15:16:27.064369 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:16:27.064352    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d129728156de9a4623be08be61bd78b2bd7f8b8da4819982e55c8d5e73c60bab\": container with ID starting with d129728156de9a4623be08be61bd78b2bd7f8b8da4819982e55c8d5e73c60bab not found: ID does not exist" containerID="d129728156de9a4623be08be61bd78b2bd7f8b8da4819982e55c8d5e73c60bab"
Sep 11 15:16:27.064413 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:27.064379    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d129728156de9a4623be08be61bd78b2bd7f8b8da4819982e55c8d5e73c60bab"} err="failed to get container status \"d129728156de9a4623be08be61bd78b2bd7f8b8da4819982e55c8d5e73c60bab\": rpc error: code = NotFound desc = could not find container \"d129728156de9a4623be08be61bd78b2bd7f8b8da4819982e55c8d5e73c60bab\": container with ID starting with d129728156de9a4623be08be61bd78b2bd7f8b8da4819982e55c8d5e73c60bab not found: ID does not exist"
Sep 11 15:16:27.076084 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:27.076064    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bdb5c7ff-4qcw9"]
Sep 11 15:16:27.081969 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:27.081948    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5bdb5c7ff-4qcw9"]
Sep 11 15:16:27.901280 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:27.901245    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bdb5c7ff-5nhqs"]
Sep 11 15:16:27.901642 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:27.901598    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80930310-9725-4223-b1ad-8f486bf0a119" containerName="manager"
Sep 11 15:16:27.901642 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:27.901609    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="80930310-9725-4223-b1ad-8f486bf0a119" containerName="manager"
Sep 11 15:16:27.901721 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:27.901673    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="80930310-9725-4223-b1ad-8f486bf0a119" containerName="manager"
Sep 11 15:16:27.901721 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:27.901682    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a38d49d7-ae58-46cd-b50c-5c103e52e5fd" containerName="istio-proxy"
Sep 11 15:16:27.913126 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:27.913098    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bdb5c7ff-5nhqs"
Sep 11 15:16:27.913851 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:27.913808    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bdb5c7ff-5nhqs"]
Sep 11 15:16:27.915499 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:27.915474    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 15:16:27.916799 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:27.916719    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 15:16:27.916799 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:27.916772    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 15:16:27.917127 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:27.917107    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sd75p\""
Sep 11 15:16:28.037943 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:28.037916    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz4z8\" (UniqueName: \"kubernetes.io/projected/e6319681-7615-4348-b3a3-8c24433882dd-kube-api-access-qz4z8\") pod \"maas-controller-5bdb5c7ff-5nhqs\" (UID: \"e6319681-7615-4348-b3a3-8c24433882dd\") " pod="opendatahub/maas-controller-5bdb5c7ff-5nhqs"
Sep 11 15:16:28.038099 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:28.037971    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6319681-7615-4348-b3a3-8c24433882dd-metrics-certs\") pod \"maas-controller-5bdb5c7ff-5nhqs\" (UID: \"e6319681-7615-4348-b3a3-8c24433882dd\") " pod="opendatahub/maas-controller-5bdb5c7ff-5nhqs"
Sep 11 15:16:28.038099 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:28.038075    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e6319681-7615-4348-b3a3-8c24433882dd-webhook-cert\") pod \"maas-controller-5bdb5c7ff-5nhqs\" (UID: \"e6319681-7615-4348-b3a3-8c24433882dd\") " pod="opendatahub/maas-controller-5bdb5c7ff-5nhqs"
Sep 11 15:16:28.139233 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:28.139202    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6319681-7615-4348-b3a3-8c24433882dd-metrics-certs\") pod \"maas-controller-5bdb5c7ff-5nhqs\" (UID: \"e6319681-7615-4348-b3a3-8c24433882dd\") " pod="opendatahub/maas-controller-5bdb5c7ff-5nhqs"
Sep 11 15:16:28.139407 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:28.139252    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e6319681-7615-4348-b3a3-8c24433882dd-webhook-cert\") pod \"maas-controller-5bdb5c7ff-5nhqs\" (UID: \"e6319681-7615-4348-b3a3-8c24433882dd\") " pod="opendatahub/maas-controller-5bdb5c7ff-5nhqs"
Sep 11 15:16:28.139407 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:28.139290    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz4z8\" (UniqueName: \"kubernetes.io/projected/e6319681-7615-4348-b3a3-8c24433882dd-kube-api-access-qz4z8\") pod \"maas-controller-5bdb5c7ff-5nhqs\" (UID: \"e6319681-7615-4348-b3a3-8c24433882dd\") " pod="opendatahub/maas-controller-5bdb5c7ff-5nhqs"
Sep 11 15:16:28.141498 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:28.141473    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e6319681-7615-4348-b3a3-8c24433882dd-webhook-cert\") pod \"maas-controller-5bdb5c7ff-5nhqs\" (UID: \"e6319681-7615-4348-b3a3-8c24433882dd\") " pod="opendatahub/maas-controller-5bdb5c7ff-5nhqs"
Sep 11 15:16:28.141582 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:28.141510    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6319681-7615-4348-b3a3-8c24433882dd-metrics-certs\") pod \"maas-controller-5bdb5c7ff-5nhqs\" (UID: \"e6319681-7615-4348-b3a3-8c24433882dd\") " pod="opendatahub/maas-controller-5bdb5c7ff-5nhqs"
Sep 11 15:16:28.146993 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:28.146976    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz4z8\" (UniqueName: \"kubernetes.io/projected/e6319681-7615-4348-b3a3-8c24433882dd-kube-api-access-qz4z8\") pod \"maas-controller-5bdb5c7ff-5nhqs\" (UID: \"e6319681-7615-4348-b3a3-8c24433882dd\") " pod="opendatahub/maas-controller-5bdb5c7ff-5nhqs"
Sep 11 15:16:28.226052 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:28.225984    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bdb5c7ff-5nhqs"
Sep 11 15:16:28.496036 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:28.496012    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bdb5c7ff-5nhqs"]
Sep 11 15:16:28.498153 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:16:28.498127    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6319681_7615_4348_b3a3_8c24433882dd.slice/crio-12c300e060f2feaa019fee39f94fb951b8ec7a572e17f4e17efa0dce302b7326 WatchSource:0}: Error finding container 12c300e060f2feaa019fee39f94fb951b8ec7a572e17f4e17efa0dce302b7326: Status 404 returned error can't find the container with id 12c300e060f2feaa019fee39f94fb951b8ec7a572e17f4e17efa0dce302b7326
Sep 11 15:16:28.662230 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:28.662194    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80930310-9725-4223-b1ad-8f486bf0a119" path="/var/lib/kubelet/pods/80930310-9725-4223-b1ad-8f486bf0a119/volumes"
Sep 11 15:16:29.067605 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:29.067569    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bdb5c7ff-5nhqs" event={"ID":"e6319681-7615-4348-b3a3-8c24433882dd","Type":"ContainerStarted","Data":"5656bdc8e4ded91cb1d89058247a208045ba901137cac8f8f730f9afd187d9e1"}
Sep 11 15:16:29.067999 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:29.067625    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bdb5c7ff-5nhqs" event={"ID":"e6319681-7615-4348-b3a3-8c24433882dd","Type":"ContainerStarted","Data":"12c300e060f2feaa019fee39f94fb951b8ec7a572e17f4e17efa0dce302b7326"}
Sep 11 15:16:29.067999 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:29.067655    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bdb5c7ff-5nhqs"
Sep 11 15:16:29.086955 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:29.086904    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5bdb5c7ff-5nhqs" podStartSLOduration=1.657322349 podStartE2EDuration="2.086886861s" totalImagesPullingTime="429.564512ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:16:29.085939357 +0000 UTC m=+2168.874203655" watchObservedRunningTime="2026-09-11 15:16:29.086886861 +0000 UTC m=+2168.875151160"
Sep 11 15:16:40.076039 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:16:40.076006    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bdb5c7ff-5nhqs"
Sep 11 15:18:46.045292 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:18:46.045256    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnwmp"]
Sep 11 15:18:46.048649 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:18:46.048632    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnwmp"
Sep 11 15:18:46.051464 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:18:46.051442    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qns29\""
Sep 11 15:18:46.067886 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:18:46.067861    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnwmp"]
Sep 11 15:18:46.133122 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:18:46.133087    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c47b6e28-b3bd-4e96-8c0a-bb016bed41dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lnwmp\" (UID: \"c47b6e28-b3bd-4e96-8c0a-bb016bed41dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnwmp"
Sep 11 15:18:46.133247 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:18:46.133144    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz6lw\" (UniqueName: \"kubernetes.io/projected/c47b6e28-b3bd-4e96-8c0a-bb016bed41dd-kube-api-access-qz6lw\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lnwmp\" (UID: \"c47b6e28-b3bd-4e96-8c0a-bb016bed41dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnwmp"
Sep 11 15:18:46.234040 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:18:46.234005    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz6lw\" (UniqueName: \"kubernetes.io/projected/c47b6e28-b3bd-4e96-8c0a-bb016bed41dd-kube-api-access-qz6lw\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lnwmp\" (UID: \"c47b6e28-b3bd-4e96-8c0a-bb016bed41dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnwmp"
Sep 11 15:18:46.234222 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:18:46.234098    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c47b6e28-b3bd-4e96-8c0a-bb016bed41dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lnwmp\" (UID: \"c47b6e28-b3bd-4e96-8c0a-bb016bed41dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnwmp"
Sep 11 15:18:46.234440 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:18:46.234422    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c47b6e28-b3bd-4e96-8c0a-bb016bed41dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lnwmp\" (UID: \"c47b6e28-b3bd-4e96-8c0a-bb016bed41dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnwmp"
Sep 11 15:18:46.243519 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:18:46.243493    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz6lw\" (UniqueName: \"kubernetes.io/projected/c47b6e28-b3bd-4e96-8c0a-bb016bed41dd-kube-api-access-qz6lw\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lnwmp\" (UID: \"c47b6e28-b3bd-4e96-8c0a-bb016bed41dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnwmp"
Sep 11 15:18:46.358391 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:18:46.358307    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnwmp"
Sep 11 15:18:46.638567 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:18:46.638544    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnwmp"]
Sep 11 15:18:46.640780 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:18:46.640751    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc47b6e28_b3bd_4e96_8c0a_bb016bed41dd.slice/crio-f56b6aa85e64896258391282e461680a28a3dacda031f2fb0291bc3c3b32a233 WatchSource:0}: Error finding container f56b6aa85e64896258391282e461680a28a3dacda031f2fb0291bc3c3b32a233: Status 404 returned error can't find the container with id f56b6aa85e64896258391282e461680a28a3dacda031f2fb0291bc3c3b32a233
Sep 11 15:18:46.712069 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:18:46.712043    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnwmp" event={"ID":"c47b6e28-b3bd-4e96-8c0a-bb016bed41dd","Type":"ContainerStarted","Data":"f56b6aa85e64896258391282e461680a28a3dacda031f2fb0291bc3c3b32a233"}
Sep 11 15:18:47.719169 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:18:47.719122    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnwmp" event={"ID":"c47b6e28-b3bd-4e96-8c0a-bb016bed41dd","Type":"ContainerStarted","Data":"5e016367cbcb23e577b7c9f287372af1d2ec2c7a4ecb2f5076b3329367b73698"}
Sep 11 15:18:47.719548 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:18:47.719200    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnwmp"
Sep 11 15:18:47.742406 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:18:47.742359    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnwmp" podStartSLOduration=1.742345132 podStartE2EDuration="1.742345132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:18:47.74007213 +0000 UTC m=+2307.528336430" watchObservedRunningTime="2026-09-11 15:18:47.742345132 +0000 UTC m=+2307.530609430"
Sep 11 15:18:58.726341 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:18:58.726300    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnwmp"
Sep 11 15:20:47.604713 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:47.604632    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x"]
Sep 11 15:20:47.608110 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:47.608091    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x"
Sep 11 15:20:47.610545 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:47.610521    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zzqqg\""
Sep 11 15:20:47.618661 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:47.618623    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x"]
Sep 11 15:20:47.729513 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:47.729476    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1973895b-f750-499c-8b56-14763a3a651c-openshift-service-ca\") pod \"maas-api-7879c6fd8-wt87x\" (UID: \"1973895b-f750-499c-8b56-14763a3a651c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x"
Sep 11 15:20:47.729690 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:47.729534    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1973895b-f750-499c-8b56-14763a3a651c-metrics-certs\") pod \"maas-api-7879c6fd8-wt87x\" (UID: \"1973895b-f750-499c-8b56-14763a3a651c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x"
Sep 11 15:20:47.729690 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:47.729668    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1973895b-f750-499c-8b56-14763a3a651c-maas-api-tls\") pod \"maas-api-7879c6fd8-wt87x\" (UID: \"1973895b-f750-499c-8b56-14763a3a651c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x"
Sep 11 15:20:47.729796 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:47.729706    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6hhb\" (UniqueName: \"kubernetes.io/projected/1973895b-f750-499c-8b56-14763a3a651c-kube-api-access-s6hhb\") pod \"maas-api-7879c6fd8-wt87x\" (UID: \"1973895b-f750-499c-8b56-14763a3a651c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x"
Sep 11 15:20:47.830346 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:47.830308    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1973895b-f750-499c-8b56-14763a3a651c-openshift-service-ca\") pod \"maas-api-7879c6fd8-wt87x\" (UID: \"1973895b-f750-499c-8b56-14763a3a651c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x"
Sep 11 15:20:47.830551 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:47.830374    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1973895b-f750-499c-8b56-14763a3a651c-metrics-certs\") pod \"maas-api-7879c6fd8-wt87x\" (UID: \"1973895b-f750-499c-8b56-14763a3a651c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x"
Sep 11 15:20:47.830551 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:47.830471    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1973895b-f750-499c-8b56-14763a3a651c-maas-api-tls\") pod \"maas-api-7879c6fd8-wt87x\" (UID: \"1973895b-f750-499c-8b56-14763a3a651c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x"
Sep 11 15:20:47.830551 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:47.830506    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6hhb\" (UniqueName: \"kubernetes.io/projected/1973895b-f750-499c-8b56-14763a3a651c-kube-api-access-s6hhb\") pod \"maas-api-7879c6fd8-wt87x\" (UID: \"1973895b-f750-499c-8b56-14763a3a651c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x"
Sep 11 15:20:47.831033 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:47.831007    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1973895b-f750-499c-8b56-14763a3a651c-openshift-service-ca\") pod \"maas-api-7879c6fd8-wt87x\" (UID: \"1973895b-f750-499c-8b56-14763a3a651c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x"
Sep 11 15:20:47.832838 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:47.832790    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1973895b-f750-499c-8b56-14763a3a651c-metrics-certs\") pod \"maas-api-7879c6fd8-wt87x\" (UID: \"1973895b-f750-499c-8b56-14763a3a651c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x"
Sep 11 15:20:47.832937 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:47.832856    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1973895b-f750-499c-8b56-14763a3a651c-maas-api-tls\") pod \"maas-api-7879c6fd8-wt87x\" (UID: \"1973895b-f750-499c-8b56-14763a3a651c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x"
Sep 11 15:20:47.838295 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:47.838270    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6hhb\" (UniqueName: \"kubernetes.io/projected/1973895b-f750-499c-8b56-14763a3a651c-kube-api-access-s6hhb\") pod \"maas-api-7879c6fd8-wt87x\" (UID: \"1973895b-f750-499c-8b56-14763a3a651c\") " pod="odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x"
Sep 11 15:20:47.920889 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:47.920788    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x"
Sep 11 15:20:48.191543 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:48.191515    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x"]
Sep 11 15:20:48.193603 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:20:48.193571    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1973895b_f750_499c_8b56_14763a3a651c.slice/crio-5304f57a6feee35eb57c4126d063eaa2e91a54d640b5cb530707280801d548d6 WatchSource:0}: Error finding container 5304f57a6feee35eb57c4126d063eaa2e91a54d640b5cb530707280801d548d6: Status 404 returned error can't find the container with id 5304f57a6feee35eb57c4126d063eaa2e91a54d640b5cb530707280801d548d6
Sep 11 15:20:48.307551 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:48.307514    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x" event={"ID":"1973895b-f750-499c-8b56-14763a3a651c","Type":"ContainerStarted","Data":"5304f57a6feee35eb57c4126d063eaa2e91a54d640b5cb530707280801d548d6"}
Sep 11 15:20:49.315283 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:49.315241    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x" event={"ID":"1973895b-f750-499c-8b56-14763a3a651c","Type":"ContainerStarted","Data":"e3095395c6c1285cd261222c16ea984010fa1876fb353bf8fe96f84e5353eb2e"}
Sep 11 15:20:49.315731 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:49.315419    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x"
Sep 11 15:20:49.332501 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:49.332453    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x" podStartSLOduration=1.778680228 podStartE2EDuration="2.33243837s" totalImagesPullingTime="553.758142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:20:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:20:49.331577314 +0000 UTC m=+2429.119841627" watchObservedRunningTime="2026-09-11 15:20:49.33243837 +0000 UTC m=+2429.120702668"
Sep 11 15:20:55.325808 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:55.325775    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7879c6fd8-wt87x"
Sep 11 15:20:55.372398 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:55.372349    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-56956d4668-hh4br"]
Sep 11 15:20:55.372629 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:55.372599    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br" podUID="802087f4-e8fa-4000-80c4-4c4a3179177d" containerName="maas-api" containerID="cri-o://6b144c06bc3113342fa33b8f6aeeab6101d2baf54917e07ce77056c62745738b" gracePeriod=30
Sep 11 15:20:56.624469 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:56.624439    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br"
Sep 11 15:20:56.710800 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:56.710729    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/802087f4-e8fa-4000-80c4-4c4a3179177d-maas-api-tls\") pod \"802087f4-e8fa-4000-80c4-4c4a3179177d\" (UID: \"802087f4-e8fa-4000-80c4-4c4a3179177d\") "
Sep 11 15:20:56.710800 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:56.710790    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/802087f4-e8fa-4000-80c4-4c4a3179177d-metrics-certs\") pod \"802087f4-e8fa-4000-80c4-4c4a3179177d\" (UID: \"802087f4-e8fa-4000-80c4-4c4a3179177d\") "
Sep 11 15:20:56.711001 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:56.710843    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms4zf\" (UniqueName: \"kubernetes.io/projected/802087f4-e8fa-4000-80c4-4c4a3179177d-kube-api-access-ms4zf\") pod \"802087f4-e8fa-4000-80c4-4c4a3179177d\" (UID: \"802087f4-e8fa-4000-80c4-4c4a3179177d\") "
Sep 11 15:20:56.711001 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:56.710875    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/802087f4-e8fa-4000-80c4-4c4a3179177d-openshift-service-ca\") pod \"802087f4-e8fa-4000-80c4-4c4a3179177d\" (UID: \"802087f4-e8fa-4000-80c4-4c4a3179177d\") "
Sep 11 15:20:56.711239 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:56.711217    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/802087f4-e8fa-4000-80c4-4c4a3179177d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "802087f4-e8fa-4000-80c4-4c4a3179177d" (UID: "802087f4-e8fa-4000-80c4-4c4a3179177d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 15:20:56.713023 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:56.712994    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/802087f4-e8fa-4000-80c4-4c4a3179177d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "802087f4-e8fa-4000-80c4-4c4a3179177d" (UID: "802087f4-e8fa-4000-80c4-4c4a3179177d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:20:56.713301 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:56.713266    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/802087f4-e8fa-4000-80c4-4c4a3179177d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "802087f4-e8fa-4000-80c4-4c4a3179177d" (UID: "802087f4-e8fa-4000-80c4-4c4a3179177d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 15:20:56.713422 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:56.713315    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/802087f4-e8fa-4000-80c4-4c4a3179177d-kube-api-access-ms4zf" (OuterVolumeSpecName: "kube-api-access-ms4zf") pod "802087f4-e8fa-4000-80c4-4c4a3179177d" (UID: "802087f4-e8fa-4000-80c4-4c4a3179177d"). InnerVolumeSpecName "kube-api-access-ms4zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 15:20:56.811719 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:56.811685    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/802087f4-e8fa-4000-80c4-4c4a3179177d-maas-api-tls\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:20:56.811719 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:56.811713    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/802087f4-e8fa-4000-80c4-4c4a3179177d-metrics-certs\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:20:56.811719 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:56.811722    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms4zf\" (UniqueName: \"kubernetes.io/projected/802087f4-e8fa-4000-80c4-4c4a3179177d-kube-api-access-ms4zf\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:20:56.812021 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:56.811732    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/802087f4-e8fa-4000-80c4-4c4a3179177d-openshift-service-ca\") on node \"ip-10-0-133-126.ec2.internal\" DevicePath \"\""
Sep 11 15:20:57.355982 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:57.355943    2568 generic.go:356] "Generic (PLEG): container finished" podID="802087f4-e8fa-4000-80c4-4c4a3179177d" containerID="6b144c06bc3113342fa33b8f6aeeab6101d2baf54917e07ce77056c62745738b" exitCode=0
Sep 11 15:20:57.356180 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:57.355998    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br"
Sep 11 15:20:57.356180 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:57.356022    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br" event={"ID":"802087f4-e8fa-4000-80c4-4c4a3179177d","Type":"ContainerDied","Data":"6b144c06bc3113342fa33b8f6aeeab6101d2baf54917e07ce77056c62745738b"}
Sep 11 15:20:57.356180 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:57.356078    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56956d4668-hh4br" event={"ID":"802087f4-e8fa-4000-80c4-4c4a3179177d","Type":"ContainerDied","Data":"cac1e7e9dca89006444d109c9e713c7183d1db9d810fc3efe540c93ffba4a88d"}
Sep 11 15:20:57.356180 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:57.356101    2568 scope.go:117] "RemoveContainer" containerID="6b144c06bc3113342fa33b8f6aeeab6101d2baf54917e07ce77056c62745738b"
Sep 11 15:20:57.365439 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:57.365421    2568 scope.go:117] "RemoveContainer" containerID="6b144c06bc3113342fa33b8f6aeeab6101d2baf54917e07ce77056c62745738b"
Sep 11 15:20:57.365681 ip-10-0-133-126 kubenswrapper[2568]: E0911 15:20:57.365655    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b144c06bc3113342fa33b8f6aeeab6101d2baf54917e07ce77056c62745738b\": container with ID starting with 6b144c06bc3113342fa33b8f6aeeab6101d2baf54917e07ce77056c62745738b not found: ID does not exist" containerID="6b144c06bc3113342fa33b8f6aeeab6101d2baf54917e07ce77056c62745738b"
Sep 11 15:20:57.365728 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:57.365684    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b144c06bc3113342fa33b8f6aeeab6101d2baf54917e07ce77056c62745738b"} err="failed to get container status \"6b144c06bc3113342fa33b8f6aeeab6101d2baf54917e07ce77056c62745738b\": rpc error: code = NotFound desc = could not find container \"6b144c06bc3113342fa33b8f6aeeab6101d2baf54917e07ce77056c62745738b\": container with ID starting with 6b144c06bc3113342fa33b8f6aeeab6101d2baf54917e07ce77056c62745738b not found: ID does not exist"
Sep 11 15:20:57.377291 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:57.377264    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-56956d4668-hh4br"]
Sep 11 15:20:57.381164 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:57.381142    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-56956d4668-hh4br"]
Sep 11 15:20:58.662089 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:20:58.662057    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="802087f4-e8fa-4000-80c4-4c4a3179177d" path="/var/lib/kubelet/pods/802087f4-e8fa-4000-80c4-4c4a3179177d/volumes"
Sep 11 15:21:17.668741 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:17.668708    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-2trt7_77a11e5d-f419-4603-ad16-f076c917205b/manager/0.log"
Sep 11 15:21:17.803607 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:17.803413    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bdb5c7ff-5nhqs_e6319681-7615-4348-b3a3-8c24433882dd/manager/0.log"
Sep 11 15:21:18.309734 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:18.309695    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f987f856-gcmcx_25614aed-8da5-40cc-a40b-e9a13d6a623a/copy-manifests/0.log"
Sep 11 15:21:18.317095 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:18.317067    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f987f856-gcmcx_25614aed-8da5-40cc-a40b-e9a13d6a623a/manager/0.log"
Sep 11 15:21:18.943726 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:18.943691    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7879c6fd8-wt87x_1973895b-f750-499c-8b56-14763a3a651c/maas-api/0.log"
Sep 11 15:21:19.308393 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:19.308361    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-nb9vs_1a473e00-62c2-4d56-acab-60d7f3026db4/postgres/0.log"
Sep 11 15:21:20.288335 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:20.288301    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-kh5nc_6f70628e-94b5-44ab-a9f0-78b4bcacdb17/manager/0.log"
Sep 11 15:21:20.687743 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:20.687666    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-lnwmp_c47b6e28-b3bd-4e96-8c0a-bb016bed41dd/manager/0.log"
Sep 11 15:21:21.523214 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:21.523123    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-x8fxp_f8e63873-4c72-46a7-ba0d-0e872730d99b/discovery/0.log"
Sep 11 15:21:21.672469 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:21.672443    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-789999dc4-rh6hh_cc4ddb02-a957-436b-8345-f0bdabe0d5d0/kube-auth-proxy/0.log"
Sep 11 15:21:22.185960 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:22.185921    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-rwvw4_0bc37496-0e43-4d25-a663-a96dd2e1f64e/payload-processing/0.log"
Sep 11 15:21:22.343835 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:22.343770    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dbb8d47db-qzqwf_80c64211-15e9-4ac3-8660-cb0ba7ae9b90/router/0.log"
Sep 11 15:21:22.710459 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:22.710424    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-4hhs5_dbfe44ff-9671-4f0d-a263-d9199c35c760/main/0.log"
Sep 11 15:21:33.155447 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:33.155418    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sxv9g_3296271a-8ecd-483d-824e-3b9896f81e3c/global-pull-secret-syncer/0.log"
Sep 11 15:21:33.281338 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:33.281301    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x9q88_e6f17c87-b5e8-4a52-a3b4-f5b409f3e75f/konnectivity-agent/0.log"
Sep 11 15:21:33.332766 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:33.332727    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-126.ec2.internal_1c006646ef20c08a2b2e3dc82507ab9d/haproxy/0.log"
Sep 11 15:21:37.746677 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:37.746580    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-kh5nc_6f70628e-94b5-44ab-a9f0-78b4bcacdb17/manager/0.log"
Sep 11 15:21:37.840550 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:37.840515    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-lnwmp_c47b6e28-b3bd-4e96-8c0a-bb016bed41dd/manager/0.log"
Sep 11 15:21:39.407594 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:39.407565    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc0805ea-2141-4f44-a021-d4521221b162/alertmanager/0.log"
Sep 11 15:21:39.432185 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:39.432153    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc0805ea-2141-4f44-a021-d4521221b162/config-reloader/0.log"
Sep 11 15:21:39.451323 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:39.451298    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc0805ea-2141-4f44-a021-d4521221b162/kube-rbac-proxy-web/0.log"
Sep 11 15:21:39.469728 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:39.469693    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc0805ea-2141-4f44-a021-d4521221b162/kube-rbac-proxy/0.log"
Sep 11 15:21:39.490022 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:39.489996    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc0805ea-2141-4f44-a021-d4521221b162/kube-rbac-proxy-metric/0.log"
Sep 11 15:21:39.508811 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:39.508779    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc0805ea-2141-4f44-a021-d4521221b162/prom-label-proxy/0.log"
Sep 11 15:21:39.531927 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:39.531906    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc0805ea-2141-4f44-a021-d4521221b162/init-config-reloader/0.log"
Sep 11 15:21:39.691216 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:39.691136    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-b2m7b_7f331a12-a765-450a-97ac-626f2f44aada/monitoring-plugin/0.log"
Sep 11 15:21:39.790827 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:39.790782    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7bk7k_2508fcba-ec05-41e2-a3ca-6490dc704fc3/node-exporter/0.log"
Sep 11 15:21:39.813266 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:39.813238    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7bk7k_2508fcba-ec05-41e2-a3ca-6490dc704fc3/kube-rbac-proxy/0.log"
Sep 11 15:21:39.831999 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:39.831978    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7bk7k_2508fcba-ec05-41e2-a3ca-6490dc704fc3/init-textfile/0.log"
Sep 11 15:21:40.174555 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:40.174522    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-vnc5j_05f6b2c3-dbe8-48a0-b786-be0e55edcb2f/prometheus-operator/0.log"
Sep 11 15:21:40.188520 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:40.188489    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-vnc5j_05f6b2c3-dbe8-48a0-b786-be0e55edcb2f/kube-rbac-proxy/0.log"
Sep 11 15:21:40.356956 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:40.356925    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f8767cffd-drhbp_97afa455-97c4-47f8-9045-15a06c0d6d10/thanos-query/0.log"
Sep 11 15:21:40.387015 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:40.386977    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f8767cffd-drhbp_97afa455-97c4-47f8-9045-15a06c0d6d10/kube-rbac-proxy-web/0.log"
Sep 11 15:21:40.410745 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:40.410646    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f8767cffd-drhbp_97afa455-97c4-47f8-9045-15a06c0d6d10/kube-rbac-proxy/0.log"
Sep 11 15:21:40.433542 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:40.433511    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f8767cffd-drhbp_97afa455-97c4-47f8-9045-15a06c0d6d10/prom-label-proxy/0.log"
Sep 11 15:21:40.456529 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:40.456496    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f8767cffd-drhbp_97afa455-97c4-47f8-9045-15a06c0d6d10/kube-rbac-proxy-rules/0.log"
Sep 11 15:21:40.482117 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:40.482087    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f8767cffd-drhbp_97afa455-97c4-47f8-9045-15a06c0d6d10/kube-rbac-proxy-metrics/0.log"
Sep 11 15:21:42.025289 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:42.025261    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-jmgrx_ec7abae9-a91a-4672-8841-c6423f247d94/networking-console-plugin/0.log"
Sep 11 15:21:43.187172 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.187141    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-2md4g_c50428a3-8505-4dff-a5f7-dfc19257b065/download-server/0.log"
Sep 11 15:21:43.272664 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.272631    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"]
Sep 11 15:21:43.273101 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.273079    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="802087f4-e8fa-4000-80c4-4c4a3179177d" containerName="maas-api"
Sep 11 15:21:43.273101 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.273103    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="802087f4-e8fa-4000-80c4-4c4a3179177d" containerName="maas-api"
Sep 11 15:21:43.273247 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.273199    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="802087f4-e8fa-4000-80c4-4c4a3179177d" containerName="maas-api"
Sep 11 15:21:43.276345 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.276328    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"
Sep 11 15:21:43.278876 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.278853    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-slnpk\"/\"openshift-service-ca.crt\""
Sep 11 15:21:43.278975 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.278889    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-slnpk\"/\"kube-root-ca.crt\""
Sep 11 15:21:43.279979 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.279961    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-slnpk\"/\"default-dockercfg-w6qf6\""
Sep 11 15:21:43.283708 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.283685    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"]
Sep 11 15:21:43.414208 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.414172    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/25c5f074-fbd6-4ce9-9421-75d6172b20fe-proc\") pod \"perf-node-gather-daemonset-tzrvx\" (UID: \"25c5f074-fbd6-4ce9-9421-75d6172b20fe\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"
Sep 11 15:21:43.414404 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.414223    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25c5f074-fbd6-4ce9-9421-75d6172b20fe-lib-modules\") pod \"perf-node-gather-daemonset-tzrvx\" (UID: \"25c5f074-fbd6-4ce9-9421-75d6172b20fe\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"
Sep 11 15:21:43.414404 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.414282    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/25c5f074-fbd6-4ce9-9421-75d6172b20fe-podres\") pod \"perf-node-gather-daemonset-tzrvx\" (UID: \"25c5f074-fbd6-4ce9-9421-75d6172b20fe\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"
Sep 11 15:21:43.414404 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.414356    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25c5f074-fbd6-4ce9-9421-75d6172b20fe-sys\") pod \"perf-node-gather-daemonset-tzrvx\" (UID: \"25c5f074-fbd6-4ce9-9421-75d6172b20fe\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"
Sep 11 15:21:43.414556 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.414423    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bttsc\" (UniqueName: \"kubernetes.io/projected/25c5f074-fbd6-4ce9-9421-75d6172b20fe-kube-api-access-bttsc\") pod \"perf-node-gather-daemonset-tzrvx\" (UID: \"25c5f074-fbd6-4ce9-9421-75d6172b20fe\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"
Sep 11 15:21:43.515667 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.515571    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25c5f074-fbd6-4ce9-9421-75d6172b20fe-sys\") pod \"perf-node-gather-daemonset-tzrvx\" (UID: \"25c5f074-fbd6-4ce9-9421-75d6172b20fe\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"
Sep 11 15:21:43.515667 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.515643    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bttsc\" (UniqueName: \"kubernetes.io/projected/25c5f074-fbd6-4ce9-9421-75d6172b20fe-kube-api-access-bttsc\") pod \"perf-node-gather-daemonset-tzrvx\" (UID: \"25c5f074-fbd6-4ce9-9421-75d6172b20fe\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"
Sep 11 15:21:43.515667 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.515668    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/25c5f074-fbd6-4ce9-9421-75d6172b20fe-proc\") pod \"perf-node-gather-daemonset-tzrvx\" (UID: \"25c5f074-fbd6-4ce9-9421-75d6172b20fe\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"
Sep 11 15:21:43.515996 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.515703    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25c5f074-fbd6-4ce9-9421-75d6172b20fe-lib-modules\") pod \"perf-node-gather-daemonset-tzrvx\" (UID: \"25c5f074-fbd6-4ce9-9421-75d6172b20fe\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"
Sep 11 15:21:43.515996 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.515704    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25c5f074-fbd6-4ce9-9421-75d6172b20fe-sys\") pod \"perf-node-gather-daemonset-tzrvx\" (UID: \"25c5f074-fbd6-4ce9-9421-75d6172b20fe\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"
Sep 11 15:21:43.515996 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.515722    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/25c5f074-fbd6-4ce9-9421-75d6172b20fe-podres\") pod \"perf-node-gather-daemonset-tzrvx\" (UID: \"25c5f074-fbd6-4ce9-9421-75d6172b20fe\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"
Sep 11 15:21:43.515996 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.515792    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/25c5f074-fbd6-4ce9-9421-75d6172b20fe-proc\") pod \"perf-node-gather-daemonset-tzrvx\" (UID: \"25c5f074-fbd6-4ce9-9421-75d6172b20fe\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"
Sep 11 15:21:43.515996 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.515860    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/25c5f074-fbd6-4ce9-9421-75d6172b20fe-podres\") pod \"perf-node-gather-daemonset-tzrvx\" (UID: \"25c5f074-fbd6-4ce9-9421-75d6172b20fe\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"
Sep 11 15:21:43.515996 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.515872    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25c5f074-fbd6-4ce9-9421-75d6172b20fe-lib-modules\") pod \"perf-node-gather-daemonset-tzrvx\" (UID: \"25c5f074-fbd6-4ce9-9421-75d6172b20fe\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"
Sep 11 15:21:43.523618 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.523598    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bttsc\" (UniqueName: \"kubernetes.io/projected/25c5f074-fbd6-4ce9-9421-75d6172b20fe-kube-api-access-bttsc\") pod \"perf-node-gather-daemonset-tzrvx\" (UID: \"25c5f074-fbd6-4ce9-9421-75d6172b20fe\") " pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"
Sep 11 15:21:43.586807 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.586762    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"
Sep 11 15:21:43.854738 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.854710    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"]
Sep 11 15:21:43.856598 ip-10-0-133-126 kubenswrapper[2568]: W0911 15:21:43.856568    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod25c5f074_fbd6_4ce9_9421_75d6172b20fe.slice/crio-70ebd194a5eb969bd01702773457b54369cae452e8e2c429f9798c5a3f675513 WatchSource:0}: Error finding container 70ebd194a5eb969bd01702773457b54369cae452e8e2c429f9798c5a3f675513: Status 404 returned error can't find the container with id 70ebd194a5eb969bd01702773457b54369cae452e8e2c429f9798c5a3f675513
Sep 11 15:21:43.897810 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:43.897784    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-zhgrn_0adfdfe5-37aa-4eac-b3fe-7d9de1053786/volume-data-source-validator/0.log"
Sep 11 15:21:44.582993 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:44.582959    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx" event={"ID":"25c5f074-fbd6-4ce9-9421-75d6172b20fe","Type":"ContainerStarted","Data":"62e068de9ead1ea86cfc96c06599d8625c2c807b16587fe6b5974761ae19f8fe"}
Sep 11 15:21:44.583409 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:44.583007    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx" event={"ID":"25c5f074-fbd6-4ce9-9421-75d6172b20fe","Type":"ContainerStarted","Data":"70ebd194a5eb969bd01702773457b54369cae452e8e2c429f9798c5a3f675513"}
Sep 11 15:21:44.583409 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:44.583038    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"
Sep 11 15:21:44.601396 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:44.601343    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx" podStartSLOduration=1.601328874 podStartE2EDuration="1.601328874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:21:44.600080779 +0000 UTC m=+2484.388345078" watchObservedRunningTime="2026-09-11 15:21:44.601328874 +0000 UTC m=+2484.389593173"
Sep 11 15:21:45.119561 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:45.119531    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x6c4c_a0d57bd1-aa8e-41b5-b3a2-720c733285f1/dns/0.log"
Sep 11 15:21:45.136623 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:45.136598    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x6c4c_a0d57bd1-aa8e-41b5-b3a2-720c733285f1/kube-rbac-proxy/0.log"
Sep 11 15:21:45.201258 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:45.201223    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xzmzt_64bef023-3ded-4c9d-9e40-1b202e960fe2/dns-node-resolver/0.log"
Sep 11 15:21:45.754587 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:45.754552    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-685db9b59b-dbkjb_9b1f55c3-70e8-4c87-a4f5-7cfb74ba05ae/registry/0.log"
Sep 11 15:21:45.823529 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:45.823504    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7mlmr_2695284a-7a7f-42ef-9f9a-e152c61edaf3/node-ca/0.log"
Sep 11 15:21:47.516810 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:47.516777    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-x8fxp_f8e63873-4c72-46a7-ba0d-0e872730d99b/discovery/0.log"
Sep 11 15:21:47.537682 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:47.537646    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-789999dc4-rh6hh_cc4ddb02-a957-436b-8345-f0bdabe0d5d0/kube-auth-proxy/0.log"
Sep 11 15:21:47.676940 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:47.676901    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-rwvw4_0bc37496-0e43-4d25-a663-a96dd2e1f64e/payload-processing/0.log"
Sep 11 15:21:47.700072 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:47.700035    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dbb8d47db-qzqwf_80c64211-15e9-4ac3-8660-cb0ba7ae9b90/router/0.log"
Sep 11 15:21:48.573163 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:48.573130    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-72l5p_d5e7d1c8-7b1d-494f-9c76-fea7135dd88c/serve-healthcheck-canary/0.log"
Sep 11 15:21:49.147645 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:49.147614    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s5plj_d8a46ee3-bfd6-45df-b4f3-191914102098/kube-rbac-proxy/0.log"
Sep 11 15:21:49.166140 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:49.166117    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s5plj_d8a46ee3-bfd6-45df-b4f3-191914102098/exporter/0.log"
Sep 11 15:21:49.185377 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:49.185348    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s5plj_d8a46ee3-bfd6-45df-b4f3-191914102098/extractor/0.log"
Sep 11 15:21:50.598472 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:50.598442    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-slnpk/perf-node-gather-daemonset-tzrvx"
Sep 11 15:21:51.816036 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:51.816002    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-2trt7_77a11e5d-f419-4603-ad16-f076c917205b/manager/0.log"
Sep 11 15:21:51.888834 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:51.888795    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bdb5c7ff-5nhqs_e6319681-7615-4348-b3a3-8c24433882dd/manager/0.log"
Sep 11 15:21:52.133768 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:52.133662    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f987f856-gcmcx_25614aed-8da5-40cc-a40b-e9a13d6a623a/manager/0.log"
Sep 11 15:21:52.152829 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:52.152800    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f987f856-gcmcx_25614aed-8da5-40cc-a40b-e9a13d6a623a/copy-manifests/0.log"
Sep 11 15:21:53.815778 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:53.815740    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-x8fxp_f8e63873-4c72-46a7-ba0d-0e872730d99b/discovery/0.log"
Sep 11 15:21:53.826355 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:53.826331    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-789999dc4-rh6hh_cc4ddb02-a957-436b-8345-f0bdabe0d5d0/kube-auth-proxy/0.log"
Sep 11 15:21:53.928694 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:53.928662    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-rwvw4_0bc37496-0e43-4d25-a663-a96dd2e1f64e/payload-processing/0.log"
Sep 11 15:21:53.945921 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:53.945882    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dbb8d47db-qzqwf_80c64211-15e9-4ac3-8660-cb0ba7ae9b90/router/0.log"
Sep 11 15:21:56.327931 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:56.327889    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-x8fxp_f8e63873-4c72-46a7-ba0d-0e872730d99b/discovery/0.log"
Sep 11 15:21:56.348440 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:56.348411    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-789999dc4-rh6hh_cc4ddb02-a957-436b-8345-f0bdabe0d5d0/kube-auth-proxy/0.log"
Sep 11 15:21:56.455131 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:56.455050    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-rwvw4_0bc37496-0e43-4d25-a663-a96dd2e1f64e/payload-processing/0.log"
Sep 11 15:21:56.475862 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:21:56.475803    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dbb8d47db-qzqwf_80c64211-15e9-4ac3-8660-cb0ba7ae9b90/router/0.log"
Sep 11 15:22:00.573316 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:22:00.573289    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nflgb_53f3bf9a-6da4-46bb-a51e-f0c10c0da500/kube-multus-additional-cni-plugins/0.log"
Sep 11 15:22:00.591003 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:22:00.590971    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nflgb_53f3bf9a-6da4-46bb-a51e-f0c10c0da500/egress-router-binary-copy/0.log"
Sep 11 15:22:00.608267 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:22:00.608237    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nflgb_53f3bf9a-6da4-46bb-a51e-f0c10c0da500/cni-plugins/0.log"
Sep 11 15:22:00.624942 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:22:00.624915    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nflgb_53f3bf9a-6da4-46bb-a51e-f0c10c0da500/bond-cni-plugin/0.log"
Sep 11 15:22:00.642040 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:22:00.642009    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nflgb_53f3bf9a-6da4-46bb-a51e-f0c10c0da500/routeoverride-cni/0.log"
Sep 11 15:22:00.659069 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:22:00.659043    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nflgb_53f3bf9a-6da4-46bb-a51e-f0c10c0da500/whereabouts-cni-bincopy/0.log"
Sep 11 15:22:00.681623 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:22:00.681573    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nflgb_53f3bf9a-6da4-46bb-a51e-f0c10c0da500/whereabouts-cni/0.log"
Sep 11 15:22:00.737994 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:22:00.737962    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hhmbs_e57ff2ae-a4ff-4f18-a25a-f0ee3627e106/kube-multus/0.log"
Sep 11 15:22:00.841114 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:22:00.840945    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zsqws_1cb940cd-c21b-4118-ad56-bb823c730604/network-metrics-daemon/0.log"
Sep 11 15:22:00.853782 ip-10-0-133-126 kubenswrapper[2568]: I0911 15:22:00.853748    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zsqws_1cb940cd-c21b-4118-ad56-bb823c730604/kube-rbac-proxy/0.log"