Sep 11 16:00:13.768299 ip-10-0-139-140 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 16:00:13.768315 ip-10-0-139-140 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 16:00:13.768325 ip-10-0-139-140 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 16:00:13.768724 ip-10-0-139-140 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 16:00:23.888036 ip-10-0-139-140 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 16:00:23.888055 ip-10-0-139-140 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 43dd2d5b230946c4bc2839c6109207eb --
Sep 11 16:02:20.042444 ip-10-0-139-140 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 16:02:20.413723 ip-10-0-139-140 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:02:20.413723 ip-10-0-139-140 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 16:02:20.413723 ip-10-0-139-140 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:02:20.413723 ip-10-0-139-140 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 16:02:20.413723 ip-10-0-139-140 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:02:20.414799 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.414706    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 16:02:20.419783 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419767    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:02:20.419783 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419783    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:02:20.419857 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419787    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:02:20.419857 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419790    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:02:20.419857 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419793    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:02:20.419857 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419796    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:02:20.419857 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419799    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:02:20.419857 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419803    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:02:20.419857 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419805    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:02:20.419857 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419808    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:02:20.419857 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419810    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:02:20.419857 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419813    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:02:20.419857 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419816    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:02:20.419857 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419819    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:02:20.419857 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419821    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:02:20.419857 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419824    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:02:20.419857 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419827    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:02:20.419857 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419830    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:02:20.419857 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419832    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:02:20.419857 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419836    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:02:20.419857 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419841    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:02:20.419857 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419845    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:02:20.420350 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419848    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:02:20.420350 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419851    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:02:20.420350 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419854    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:02:20.420350 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419857    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:02:20.420350 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419860    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:02:20.420350 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419863    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:02:20.420350 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419866    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:02:20.420350 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419869    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:02:20.420350 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419873    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:02:20.420350 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419875    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:02:20.420350 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419878    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:02:20.420350 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419881    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:02:20.420350 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419885    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:02:20.420350 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419888    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:02:20.420350 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419891    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:02:20.420350 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419894    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:02:20.420350 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419896    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:02:20.420350 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419899    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:02:20.420350 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419902    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:02:20.420822 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419905    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:02:20.420822 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419908    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:02:20.420822 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419910    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:02:20.420822 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419913    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:02:20.420822 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419915    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:02:20.420822 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419918    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:02:20.420822 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419920    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:02:20.420822 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419924    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:02:20.420822 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419927    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:02:20.420822 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419930    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:02:20.420822 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419932    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:02:20.420822 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419934    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:02:20.420822 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419937    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:02:20.420822 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419941    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:02:20.420822 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419945    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:02:20.420822 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419947    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:02:20.420822 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419950    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:02:20.420822 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419952    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:02:20.420822 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419955    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:02:20.420822 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419957    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:02:20.421316 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419960    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:02:20.421316 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419963    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:02:20.421316 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419965    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:02:20.421316 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419968    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:02:20.421316 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419970    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:02:20.421316 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419973    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:02:20.421316 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419975    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:02:20.421316 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419978    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:02:20.421316 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419980    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:02:20.421316 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419983    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:02:20.421316 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419986    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:02:20.421316 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419988    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:02:20.421316 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419991    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:02:20.421316 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419993    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:02:20.421316 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419996    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:02:20.421316 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.419998    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:02:20.421316 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420001    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:02:20.421316 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420003    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:02:20.421316 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420005    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:02:20.421316 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420008    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:02:20.421785 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420010    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:02:20.421785 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420016    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:02:20.421785 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420018    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:02:20.421785 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420021    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:02:20.421785 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420023    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:02:20.421785 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420025    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:02:20.421785 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420028    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:02:20.421785 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420045    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:02:20.421785 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420048    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:02:20.421785 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420050    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:02:20.421785 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420054    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:02:20.421785 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420056    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:02:20.421785 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420059    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:02:20.421785 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420062    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:02:20.421785 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420064    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:02:20.421785 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420067    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:02:20.421785 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420069    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:02:20.421785 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420072    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:02:20.421785 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420854    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:02:20.421785 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420866    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:02:20.422379 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420870    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:02:20.422379 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420874    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:02:20.422379 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420877    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:02:20.422379 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420880    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:02:20.422379 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420883    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:02:20.422379 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420886    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:02:20.422379 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420889    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:02:20.422379 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420892    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:02:20.422379 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420894    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:02:20.422379 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420897    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:02:20.422379 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420899    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:02:20.422379 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420902    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:02:20.422379 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420905    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:02:20.422379 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420907    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:02:20.422379 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420910    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:02:20.422379 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420913    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:02:20.422379 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420915    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:02:20.422379 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420918    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:02:20.422379 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420920    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:02:20.422379 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420923    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:02:20.422899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420925    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:02:20.422899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420928    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:02:20.422899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420931    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:02:20.422899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420933    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:02:20.422899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420935    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:02:20.422899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420938    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:02:20.422899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420940    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:02:20.422899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420943    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:02:20.422899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420945    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:02:20.422899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420947    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:02:20.422899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420950    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:02:20.422899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420954    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:02:20.422899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420957    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:02:20.422899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420959    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:02:20.422899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420962    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:02:20.422899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420964    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:02:20.422899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420967    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:02:20.422899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420969    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:02:20.422899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420972    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:02:20.422899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420975    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:02:20.423407 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420977    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:02:20.423407 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420980    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:02:20.423407 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420982    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:02:20.423407 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420985    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:02:20.423407 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420987    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:02:20.423407 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420990    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:02:20.423407 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420992    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:02:20.423407 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420995    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:02:20.423407 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.420997    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:02:20.423407 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421000    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:02:20.423407 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421005    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:02:20.423407 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421009    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:02:20.423407 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421012    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:02:20.423407 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421015    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:02:20.423407 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421019    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:02:20.423407 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421021    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:02:20.423407 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421024    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:02:20.423407 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421027    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:02:20.423407 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421030    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:02:20.423899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421046    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:02:20.423899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421049    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:02:20.423899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421053    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:02:20.423899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421057    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:02:20.423899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421061    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:02:20.423899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421067    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:02:20.423899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421070    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:02:20.423899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421073    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:02:20.423899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421076    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:02:20.423899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421079    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:02:20.423899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421082    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:02:20.423899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421085    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:02:20.423899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421088    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:02:20.423899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421090    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:02:20.423899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421093    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:02:20.423899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421095    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:02:20.423899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421098    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:02:20.423899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421100    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:02:20.423899 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421103    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:02:20.424393 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421106    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:02:20.424393 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421108    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:02:20.424393 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421111    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:02:20.424393 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421113    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:02:20.424393 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421116    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:02:20.424393 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421119    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:02:20.424393 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421121    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:02:20.424393 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421123    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:02:20.424393 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421126    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:02:20.424393 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421128    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:02:20.424393 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421131    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:02:20.424393 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421133    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:02:20.424393 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421135    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:02:20.424393 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421138    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:02:20.424393 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421140    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:02:20.424393 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421143    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:02:20.424393 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421146    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:02:20.424393 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421149    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:02:20.424393 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421151    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:02:20.424393 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421563    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:02:20.424883 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421568    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:02:20.424883 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421572    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:02:20.424883 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421575    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:02:20.424883 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421578    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:02:20.424883 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421581    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:02:20.424883 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421584    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:02:20.424883 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421587    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:02:20.424883 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421589    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:02:20.424883 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421592    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:02:20.424883 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421594    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:02:20.424883 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421597    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:02:20.424883 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421599    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:02:20.424883 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421602    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:02:20.424883 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421604    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:02:20.424883 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421607    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:02:20.424883 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421609    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:02:20.424883 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421612    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:02:20.424883 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421616    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:02:20.424883 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421618    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:02:20.424883 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421620    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:02:20.425406 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421623    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:02:20.425406 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421625    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:02:20.425406 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421628    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:02:20.425406 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421630    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:02:20.425406 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421632    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:02:20.425406 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421635    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:02:20.425406 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421637    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:02:20.425406 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421640    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:02:20.425406 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421643    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:02:20.425406 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421648    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:02:20.425406 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421651    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:02:20.425406 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421654    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:02:20.425406 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421657    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:02:20.425406 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421660    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:02:20.425406 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421662    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:02:20.425406 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421665    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:02:20.425406 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421668    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:02:20.425406 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421670    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:02:20.425406 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421673    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:02:20.425874 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421675    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:02:20.425874 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421678    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:02:20.425874 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421681    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:02:20.425874 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421683    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:02:20.425874 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421686    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:02:20.425874 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421690    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:02:20.425874 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421693    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:02:20.425874 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421696    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:02:20.425874 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421699    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:02:20.425874 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421701    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:02:20.425874 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421704    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:02:20.425874 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421707    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:02:20.425874 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421711    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:02:20.425874 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421714    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:02:20.425874 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421717    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:02:20.425874 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421720    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:02:20.425874 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421723    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:02:20.425874 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421725    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:02:20.425874 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421728    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:02:20.426360 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421732    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:02:20.426360 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421736    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:02:20.426360 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421738    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:02:20.426360 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421742    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:02:20.426360 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421745    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:02:20.426360 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421747    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:02:20.426360 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421750    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:02:20.426360 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421753    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:02:20.426360 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421755    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:02:20.426360 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421757    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:02:20.426360 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421760    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:02:20.426360 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421762    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:02:20.426360 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421765    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:02:20.426360 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421767    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:02:20.426360 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421770    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:02:20.426360 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421773    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:02:20.426360 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421775    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:02:20.426360 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421778    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:02:20.426360 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421780    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:02:20.426360 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421783    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:02:20.426886 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421785    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:02:20.426886 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421788    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:02:20.426886 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421790    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:02:20.426886 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421793    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:02:20.426886 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421796    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:02:20.426886 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421799    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:02:20.426886 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421801    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:02:20.426886 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421804    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:02:20.426886 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421806    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:02:20.426886 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421808    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:02:20.426886 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421811    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:02:20.426886 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421813    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:02:20.426886 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421816    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:02:20.426886 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421818    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:02:20.426886 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421821    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:02:20.426886 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421824    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:02:20.426886 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421826    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:02:20.426886 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421829    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:02:20.426886 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421831    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:02:20.426886 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.421834    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.421910    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.421951    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.421960    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.421965    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.421970    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.421973    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.421977    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.421981    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.421985    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.421987    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.421991    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.421994    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.421997    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422000    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422003    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422006    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422009    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422012    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422017    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422019    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422022    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422025    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422028    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 16:02:20.427399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422048    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422053    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422057    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422062    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422065    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422068    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422071    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422074    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422077    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422082    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422085    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422088    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422091    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422094    2577 flags.go:64] FLAG: --enable-server="true"
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422097    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422102    2577 flags.go:64] FLAG: --event-burst="100"
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422104    2577 flags.go:64] FLAG: --event-qps="50"
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422107    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422110    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422113    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422117    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422120    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422123    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422126    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422129    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 16:02:20.427976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422131    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422135    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422138    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422140    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422143    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422146    2577 flags.go:64] FLAG: --feature-gates=""
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422149    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422152    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422155    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422158    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422161    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422164    2577 flags.go:64] FLAG: --help="false"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422166    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-139-140.ec2.internal"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422169    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422172    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422175    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422178    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422184    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422186    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422189    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422192    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422195    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422197    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422201    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 16:02:20.428620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422204    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422206    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422209    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422212    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422215    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422217    2577 flags.go:64] FLAG: --lock-file=""
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422221    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422223    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422227    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422232    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422235    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422238    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422240    2577 flags.go:64] FLAG: --logging-format="text"
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422243    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422246    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422249    2577 flags.go:64] FLAG: --manifest-url=""
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422252    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422256    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422259    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422263    2577 flags.go:64] FLAG: --max-pods="110"
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422266    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422269    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422271    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422274    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422278    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 16:02:20.429245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422281    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422284    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422292    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422295    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422297    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422300    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422303    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422308    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422311    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422314    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422319    2577 flags.go:64] FLAG: --port="10250"
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422322    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422325    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b0336b28b0a3175c"
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422328    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422331    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422334    2577 flags.go:64] FLAG: --register-node="true"
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422337    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422341    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422343    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422346    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422349    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422352    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422355    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422358    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422361    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 16:02:20.429863 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422364    2577 flags.go:64] FLAG: --runonce="false"
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422366    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422369    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422372    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422375    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422378    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422381    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422384    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422387    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422390    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422393    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422396    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422399    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422402    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422405    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422407    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422413    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422418    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422421    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422425    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422429    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422432    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422435    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422438    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422441    2577 flags.go:64] FLAG: --v="2"
Sep 11 16:02:20.430537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422445    2577 flags.go:64] FLAG: --version="false"
Sep 11 16:02:20.431157 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422449    2577 flags.go:64] FLAG: --vmodule=""
Sep 11 16:02:20.431157 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422453    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 16:02:20.431157 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.422456    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 16:02:20.431157 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422585    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:02:20.431157 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422589    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:02:20.431157 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422593    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:02:20.431157 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422595    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:02:20.431157 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422598    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:02:20.431157 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422601    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:02:20.431157 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422604    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:02:20.431157 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422607    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:02:20.431157 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422610    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:02:20.431157 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422613    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:02:20.431157 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422615    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:02:20.431157 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422618    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:02:20.431157 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422620    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:02:20.431157 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422623    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:02:20.431157 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422627    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:02:20.431157 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422629    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:02:20.431157 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422632    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:02:20.431647 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422634    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:02:20.431647 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422637    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:02:20.431647 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422640    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:02:20.431647 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422647    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:02:20.431647 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422650    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:02:20.431647 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422653    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:02:20.431647 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422656    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:02:20.431647 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422659    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:02:20.431647 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422662    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:02:20.431647 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422664    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:02:20.431647 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422667    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:02:20.431647 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422670    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:02:20.431647 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422673    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:02:20.431647 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422675    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:02:20.431647 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422678    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:02:20.431647 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422680    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:02:20.431647 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422683    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:02:20.431647 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422686    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:02:20.431647 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422688    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:02:20.432143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422691    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:02:20.432143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422693    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:02:20.432143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422696    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:02:20.432143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422698    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:02:20.432143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422701    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:02:20.432143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422704    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:02:20.432143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422707    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:02:20.432143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422709    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:02:20.432143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422712    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:02:20.432143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422714    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:02:20.432143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422717    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:02:20.432143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422720    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:02:20.432143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422723    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:02:20.432143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422727    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:02:20.432143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422730    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:02:20.432143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422733    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:02:20.432143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422737    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:02:20.432143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422740    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:02:20.432143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422743    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:02:20.432143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422747    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:02:20.432645 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422750    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:02:20.432645 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422753    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:02:20.432645 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422755    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:02:20.432645 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422758    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:02:20.432645 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422761    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:02:20.432645 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422764    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:02:20.432645 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422766    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:02:20.432645 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422769    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:02:20.432645 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422771    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:02:20.432645 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422774    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:02:20.432645 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422776    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:02:20.432645 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422779    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:02:20.432645 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422783    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:02:20.432645 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422787    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:02:20.432645 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422789    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:02:20.432645 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422792    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:02:20.432645 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422795    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:02:20.432645 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422798    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:02:20.432645 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422800    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:02:20.432645 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422803    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:02:20.433147 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422806    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:02:20.433147 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422808    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:02:20.433147 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422810    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:02:20.433147 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422813    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:02:20.433147 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422816    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:02:20.433147 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422818    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:02:20.433147 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422821    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:02:20.433147 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422824    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:02:20.433147 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422828    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:02:20.433147 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422830    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:02:20.433147 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422833    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:02:20.433147 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422836    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:02:20.433147 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422839    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:02:20.433147 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422841    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:02:20.433147 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422844    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:02:20.433147 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422847    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:02:20.433147 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422849    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:02:20.433147 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422852    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:02:20.433147 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422854    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:02:20.433147 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422857    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:02:20.433666 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422860    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:02:20.433666 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422862    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:02:20.433666 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.422865    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:02:20.433666 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.423361    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:02:20.433666 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.430218    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 16:02:20.433666 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.430235    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 16:02:20.433666 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.430959    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 16:02:20.433666 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.430966    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 16:02:20.433666 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431027    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:02:20.433666 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431049    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:02:20.433666 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431053    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:02:20.433666 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431056    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:02:20.433666 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431060    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:02:20.433666 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431062    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:02:20.433666 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431065    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:02:20.434058 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431068    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:02:20.434058 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431072    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:02:20.434058 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431075    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:02:20.434058 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431078    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:02:20.434058 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431081    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:02:20.434058 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431083    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:02:20.434058 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431086    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:02:20.434058 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431089    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:02:20.434058 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431092    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:02:20.434058 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431094    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:02:20.434058 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431097    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:02:20.434058 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431099    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:02:20.434058 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431102    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:02:20.434058 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431104    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:02:20.434058 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431107    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:02:20.434058 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431109    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:02:20.434058 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431112    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:02:20.434058 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431115    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:02:20.434058 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431118    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:02:20.434522 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431120    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:02:20.434522 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431122    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:02:20.434522 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431125    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:02:20.434522 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431128    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:02:20.434522 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431131    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:02:20.434522 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431134    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:02:20.434522 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431138    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:02:20.434522 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431142    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:02:20.434522 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431145    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:02:20.434522 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431148    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:02:20.434522 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431151    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:02:20.434522 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431153    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:02:20.434522 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431156    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:02:20.434522 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431158    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:02:20.434522 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431162    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:02:20.434522 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431165    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:02:20.434522 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431168    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:02:20.434522 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431170    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:02:20.434522 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431173    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:02:20.434983 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431176    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:02:20.434983 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431178    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:02:20.434983 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431181    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:02:20.434983 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431183    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:02:20.434983 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431186    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:02:20.434983 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431189    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:02:20.434983 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431191    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:02:20.434983 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431194    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:02:20.434983 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431196    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:02:20.434983 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431199    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:02:20.434983 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431202    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:02:20.434983 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431204    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:02:20.434983 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431206    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:02:20.434983 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431209    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:02:20.434983 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431212    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:02:20.434983 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431214    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:02:20.434983 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431218    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:02:20.434983 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431220    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:02:20.434983 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431223    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:02:20.434983 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431226    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:02:20.435480 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431228    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:02:20.435480 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431231    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:02:20.435480 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431233    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:02:20.435480 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431236    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:02:20.435480 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431238    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:02:20.435480 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431241    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:02:20.435480 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431244    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:02:20.435480 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431246    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:02:20.435480 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431249    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:02:20.435480 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431251    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:02:20.435480 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431254    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:02:20.435480 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431258    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:02:20.435480 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431261    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:02:20.435480 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431264    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:02:20.435480 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431267    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:02:20.435480 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431269    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:02:20.435480 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431271    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:02:20.435480 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431274    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:02:20.435480 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431276    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:02:20.435936 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431280    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:02:20.435936 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431283    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:02:20.435936 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431287    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:02:20.435936 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431290    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:02:20.435936 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431292    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:02:20.435936 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431295    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:02:20.435936 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431298    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:02:20.435936 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431301    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:02:20.435936 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431304    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:02:20.435936 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431307    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:02:20.435936 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431310    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:02:20.435936 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431312    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:02:20.435936 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431315    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:02:20.435936 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431318    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:02:20.435936 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431321    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:02:20.436311 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.431325    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:02:20.436311 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431435    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:02:20.436311 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431439    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:02:20.436311 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431442    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:02:20.436311 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431445    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:02:20.436311 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431448    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:02:20.436311 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431451    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:02:20.436311 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431454    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:02:20.436311 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431457    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:02:20.436311 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431459    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:02:20.436311 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431463    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:02:20.436311 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431467    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:02:20.436311 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431470    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:02:20.436311 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431473    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:02:20.436311 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431476    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:02:20.436681 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431479    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:02:20.436681 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431481    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:02:20.436681 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431484    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:02:20.436681 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431487    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:02:20.436681 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431489    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:02:20.436681 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431491    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:02:20.436681 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431494    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:02:20.436681 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431496    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:02:20.436681 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431499    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:02:20.436681 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431501    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:02:20.436681 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431504    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:02:20.436681 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431507    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:02:20.436681 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431510    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:02:20.436681 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431512    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:02:20.436681 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431515    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:02:20.436681 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431517    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:02:20.436681 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431520    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:02:20.436681 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431522    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:02:20.436681 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431524    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:02:20.436681 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431527    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:02:20.437333 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431530    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:02:20.437333 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431532    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:02:20.437333 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431534    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:02:20.437333 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431537    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:02:20.437333 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431539    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:02:20.437333 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431542    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:02:20.437333 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431545    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:02:20.437333 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431547    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:02:20.437333 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431550    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:02:20.437333 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431552    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:02:20.437333 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431554    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:02:20.437333 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431558    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:02:20.437333 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431561    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:02:20.437333 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431564    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:02:20.437333 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431567    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:02:20.437333 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431570    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:02:20.437333 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431573    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:02:20.437333 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431576    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:02:20.437333 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431578    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:02:20.437790 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431581    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:02:20.437790 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431583    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:02:20.437790 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431585    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:02:20.437790 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431588    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:02:20.437790 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431591    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:02:20.437790 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431594    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:02:20.437790 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431597    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:02:20.437790 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431617    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:02:20.437790 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431621    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:02:20.437790 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431624    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:02:20.437790 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431628    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:02:20.437790 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431631    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:02:20.437790 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431634    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:02:20.437790 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431637    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:02:20.437790 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431639    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:02:20.437790 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431642    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:02:20.437790 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431644    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:02:20.437790 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431647    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:02:20.437790 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431649    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:02:20.437790 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431653    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:02:20.438288 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431656    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:02:20.438288 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431660    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:02:20.438288 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431664    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:02:20.438288 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431666    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:02:20.438288 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431669    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:02:20.438288 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431671    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:02:20.438288 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431674    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:02:20.438288 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431677    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:02:20.438288 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431679    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:02:20.438288 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431681    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:02:20.438288 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431684    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:02:20.438288 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431686    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:02:20.438288 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431688    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:02:20.438288 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431691    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:02:20.438288 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431693    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:02:20.438288 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431696    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:02:20.438288 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431698    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:02:20.438288 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431701    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:02:20.438288 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431704    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:02:20.438288 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431707    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:02:20.438774 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431709    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:02:20.438774 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431711    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:02:20.438774 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431714    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:02:20.438774 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431716    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:02:20.438774 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431718    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:02:20.438774 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:20.431721    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:02:20.438774 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.431725    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:02:20.438774 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.431834    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 16:02:20.438774 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.433970    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 16:02:20.438774 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.434899    2577 server.go:1019] "Starting client certificate rotation"
Sep 11 16:02:20.438774 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.435001    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 16:02:20.438774 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.435026    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 16:02:20.438774 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.437291    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 16:02:20.441804 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.441778    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 16:02:20.454140 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.454115    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 11 16:02:20.457142 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.457125    2577 log.go:25] "Validated CRI v1 image API"
Sep 11 16:02:20.457984 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.457971    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 16:02:20.460961 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.460942    2577 fs.go:135] Filesystem UUIDs: map[41b77094-1b20-45d2-b970-b8ba03e36b86:/dev/nvme0n1p3 4ab15d1c-9578-4545-88dc-738814eae78c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 11 16:02:20.461015 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.460962    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 16:02:20.466154 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.466016    2577 manager.go:216] Machine: {Timestamp:2026-09-11 16:02:20.4648701 +0000 UTC m=+0.310513646 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099211 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fd89c6185f87ae60b92ff48ed15c7 SystemUUID:ec2fd89c-6185-f87a-e60b-92ff48ed15c7 BootID:43dd2d5b-2309-46c4-bc28-39c6109207eb Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f9:62:fd:f5:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f9:62:fd:f5:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:f4:9c:d2:5b:4a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 16:02:20.466154 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.466151    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 16:02:20.466286 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.466241    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 16:02:20.468533 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.468517    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 16:02:20.468945 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.468921    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 16:02:20.469110 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.468951    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-140.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 16:02:20.469649 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.469639    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 16:02:20.469681 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.469652    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 16:02:20.469681 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.469660    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 16:02:20.470281 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.470269    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 16:02:20.470937 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.470927    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 16:02:20.474915 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.474901    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 16:02:20.475061 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.475050    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 16:02:20.475101 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.475087    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 16:02:20.475101 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.475099    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 16:02:20.475152 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.475113    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 16:02:20.475152 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.475123    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 11 16:02:20.475152 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.475132    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 16:02:20.475559 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.475547    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 16:02:20.475595 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.475569    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 16:02:20.478157 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.478138    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 16:02:20.479950 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.479937    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 16:02:20.479999 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.479967    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 16:02:20.481303 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.481283    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 16:02:20.481303 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.481306    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 16:02:20.481478 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.481316    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 16:02:20.481478 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.481322    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 16:02:20.481478 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.481334    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 16:02:20.481478 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.481342    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 16:02:20.481478 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.481355    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 16:02:20.481478 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.481360    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 16:02:20.481478 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.481368    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 16:02:20.481478 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.481373    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 16:02:20.481478 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.481381    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 16:02:20.481478 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.481386    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 16:02:20.482077 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.482066    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 16:02:20.482077 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.482077    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 16:02:20.486498 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.486468    2577 server.go:1268] "Started kubelet"
Sep 11 16:02:20.486854 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.486806    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 16:02:20.486954 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.486814    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 16:02:20.486954 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.486908    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 16:02:20.487802 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.487719    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-140.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:02:20.487947 ip-10-0-139-140 systemd[1]: Started Kubernetes Kubelet.
Sep 11 16:02:20.488741 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:20.488414    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 16:02:20.488741 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:20.488449    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-140.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 16:02:20.489161 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.489140    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 16:02:20.490761 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.490739    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 16:02:20.491294 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:20.491266    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 16:02:20.493090 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.493072    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9pstj"
Sep 11 16:02:20.493181 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:20.492338    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-140.ec2.internal.18d44fa94a3ba35c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-140.ec2.internal,UID:ip-10-0-139-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-140.ec2.internal,},FirstTimestamp:2026-09-11 16:02:20.486419292 +0000 UTC m=+0.332062839,LastTimestamp:2026-09-11 16:02:20.486419292 +0000 UTC m=+0.332062839,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-140.ec2.internal,}"
Sep 11 16:02:20.494413 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.494400    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 16:02:20.494776 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.494760    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 16:02:20.494920 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:20.494898    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-140.ec2.internal\" not found"
Sep 11 16:02:20.495251 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.495233    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 16:02:20.495251 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.495237    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 16:02:20.495355 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.495259    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 16:02:20.495436 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.495424    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 16:02:20.495495 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.495436    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 16:02:20.495787 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.495775    2577 factory.go:153] Registering CRI-O factory
Sep 11 16:02:20.495868 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.495790    2577 factory.go:223] Registration of the crio container factory successfully
Sep 11 16:02:20.495868 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.495841    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 16:02:20.495868 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.495850    2577 factory.go:55] Registering systemd factory
Sep 11 16:02:20.495868 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.495859    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 11 16:02:20.496068 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.495874    2577 factory.go:103] Registering Raw factory
Sep 11 16:02:20.496068 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.495888    2577 manager.go:1196] Started watching for new ooms in manager
Sep 11 16:02:20.496348 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.496334    2577 manager.go:349] Starting recovery of all containers
Sep 11 16:02:20.501007 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.500631    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9pstj"
Sep 11 16:02:20.505798 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:20.505770    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-140.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 16:02:20.505909 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:20.505842    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 16:02:20.506241 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.506225    2577 manager.go:354] Recovery completed
Sep 11 16:02:20.512474 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.512458    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:02:20.515349 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.515328    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:02:20.515417 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.515367    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:02:20.515417 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.515378    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:02:20.515954 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.515936    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 16:02:20.515954 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.515952    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 16:02:20.516072 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.515970    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 16:02:20.516072 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.515953    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:02:20.516716 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.516703    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:02:20.516783 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.516726    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:02:20.516783 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.516735    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:02:20.517916 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.517905    2577 policy_none.go:49] "None policy: Start"
Sep 11 16:02:20.517959 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.517920    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 16:02:20.517959 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.517930    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 16:02:20.520045 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.520019    2577 policy_none.go:47] "Start"
Sep 11 16:02:20.573768 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.560682    2577 manager.go:337] "Starting Device Plugin manager"
Sep 11 16:02:20.573768 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:20.560721    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 16:02:20.573768 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.560731    2577 server.go:87] "Starting device plugin registration server"
Sep 11 16:02:20.573768 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.561145    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 16:02:20.573768 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.561156    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 16:02:20.573768 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.561254    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 16:02:20.573768 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.561334    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 16:02:20.573768 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.561340    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 16:02:20.573768 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:20.562310    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 16:02:20.573768 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:20.562358    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-140.ec2.internal\" not found"
Sep 11 16:02:20.624241 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.624195    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 16:02:20.625590 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.625571    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 16:02:20.625667 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.625631    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 16:02:20.625667 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.625655    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 16:02:20.625778 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:20.625700    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 16:02:20.628493 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.628476    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:02:20.661631 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.661602    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:02:20.663576 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.663342    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:02:20.663576 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.663381    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:02:20.663576 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.663397    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:02:20.663576 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.663429    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-140.ec2.internal"
Sep 11 16:02:20.669786 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.669697    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-140.ec2.internal"
Sep 11 16:02:20.669786 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:20.669726    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-140.ec2.internal\": node \"ip-10-0-139-140.ec2.internal\" not found"
Sep 11 16:02:20.685404 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:20.685376    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-140.ec2.internal\" not found"
Sep 11 16:02:20.726795 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.726747    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-140.ec2.internal"]
Sep 11 16:02:20.726975 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.726859    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:02:20.727844 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.727827    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:02:20.727946 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.727864    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:02:20.727946 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.727880    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:02:20.730195 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.730179    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:02:20.730316 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.730300    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal"
Sep 11 16:02:20.730365 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.730331    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:02:20.731001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.730982    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:02:20.731111 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.731013    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:02:20.731111 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.731022    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:02:20.731111 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.730991    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:02:20.731111 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.731074    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:02:20.731111 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.731084    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:02:20.733272 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.733255    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.ec2.internal"
Sep 11 16:02:20.733341 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.733287    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:02:20.733984 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.733971    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:02:20.734062 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.733990    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:02:20.734062 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.734003    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:02:20.751647 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:20.751621    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-140.ec2.internal\" not found" node="ip-10-0-139-140.ec2.internal"
Sep 11 16:02:20.755089 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:20.755073    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-140.ec2.internal\" not found" node="ip-10-0-139-140.ec2.internal"
Sep 11 16:02:20.786228 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:20.786198    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-140.ec2.internal\" not found"
Sep 11 16:02:20.797248 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.797224    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c6d1b33687c0d99c8bd06c017ca0016-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal\" (UID: \"0c6d1b33687c0d99c8bd06c017ca0016\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal"
Sep 11 16:02:20.797329 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.797253    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/615418bae492da132dc002c225bf9501-config\") pod \"kube-apiserver-proxy-ip-10-0-139-140.ec2.internal\" (UID: \"615418bae492da132dc002c225bf9501\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.ec2.internal"
Sep 11 16:02:20.797329 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.797271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c6d1b33687c0d99c8bd06c017ca0016-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal\" (UID: \"0c6d1b33687c0d99c8bd06c017ca0016\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal"
Sep 11 16:02:20.886659 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:20.886624    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-140.ec2.internal\" not found"
Sep 11 16:02:20.898108 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.898086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c6d1b33687c0d99c8bd06c017ca0016-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal\" (UID: \"0c6d1b33687c0d99c8bd06c017ca0016\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal"
Sep 11 16:02:20.898234 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.898116    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c6d1b33687c0d99c8bd06c017ca0016-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal\" (UID: \"0c6d1b33687c0d99c8bd06c017ca0016\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal"
Sep 11 16:02:20.898234 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.898134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/615418bae492da132dc002c225bf9501-config\") pod \"kube-apiserver-proxy-ip-10-0-139-140.ec2.internal\" (UID: \"615418bae492da132dc002c225bf9501\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.ec2.internal"
Sep 11 16:02:20.898234 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.898211    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/615418bae492da132dc002c225bf9501-config\") pod \"kube-apiserver-proxy-ip-10-0-139-140.ec2.internal\" (UID: \"615418bae492da132dc002c225bf9501\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.ec2.internal"
Sep 11 16:02:20.898234 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.898214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c6d1b33687c0d99c8bd06c017ca0016-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal\" (UID: \"0c6d1b33687c0d99c8bd06c017ca0016\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal"
Sep 11 16:02:20.898358 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:20.898223    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c6d1b33687c0d99c8bd06c017ca0016-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal\" (UID: \"0c6d1b33687c0d99c8bd06c017ca0016\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal"
Sep 11 16:02:20.987591 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:20.987516    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-140.ec2.internal\" not found"
Sep 11 16:02:21.054009 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:21.053975    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal"
Sep 11 16:02:21.057101 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:21.057084    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.ec2.internal"
Sep 11 16:02:21.088639 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:21.088604    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-140.ec2.internal\" not found"
Sep 11 16:02:21.189112 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:21.189069    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-140.ec2.internal\" not found"
Sep 11 16:02:21.289685 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:21.289615    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-140.ec2.internal\" not found"
Sep 11 16:02:21.390161 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:21.390125    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-140.ec2.internal\" not found"
Sep 11 16:02:21.412276 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:21.412247    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:02:21.434504 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:21.434471    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 16:02:21.434958 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:21.434619    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 16:02:21.434958 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:21.434660    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 16:02:21.490623 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:21.490595    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-140.ec2.internal\" not found"
Sep 11 16:02:21.505414 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:21.505379    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 15:57:20 +0000 UTC" deadline="2028-02-12 16:00:09.285576349 +0000 UTC"
Sep 11 16:02:21.505414 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:21.505405    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12455h57m47.780173861s"
Sep 11 16:02:21.516182 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:21.516153    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:02:21.591133 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:21.591020    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-140.ec2.internal\" not found"
Sep 11 16:02:21.691588 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:21.691550    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-140.ec2.internal\" not found"
Sep 11 16:02:21.762440 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:21.762395    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:02:21.785928 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:21.785879    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6d1b33687c0d99c8bd06c017ca0016.slice/crio-a9c83277aef6ed9e8629cacabdee71f47df0da5f4d0a3c8bde90a1ae2e931453 WatchSource:0}: Error finding container a9c83277aef6ed9e8629cacabdee71f47df0da5f4d0a3c8bde90a1ae2e931453: Status 404 returned error can't find the container with id a9c83277aef6ed9e8629cacabdee71f47df0da5f4d0a3c8bde90a1ae2e931453
Sep 11 16:02:21.789209 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:21.789192    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:02:21.795073 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:21.795051    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal"
Sep 11 16:02:21.808403 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:21.808376    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 16:02:21.809479 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:21.809464    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.ec2.internal"
Sep 11 16:02:21.819412 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:21.819387    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 16:02:22.126313 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:22.126277    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615418bae492da132dc002c225bf9501.slice/crio-25867f28953067922ac810b74518e91a87f3f20ff781ba3098af2dee4f535df8 WatchSource:0}: Error finding container 25867f28953067922ac810b74518e91a87f3f20ff781ba3098af2dee4f535df8: Status 404 returned error can't find the container with id 25867f28953067922ac810b74518e91a87f3f20ff781ba3098af2dee4f535df8
Sep 11 16:02:22.476370 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.476279    2577 apiserver.go:52] "Watching apiserver"
Sep 11 16:02:22.484783 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.484747    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 16:02:22.485871 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.485840    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-dn4s5","openshift-multus/network-metrics-daemon-vw4vl","openshift-network-diagnostics/network-check-target-8j7q7","openshift-network-operator/iptables-alerter-pdmrf","openshift-ovn-kubernetes/ovnkube-node-ng2ww","kube-system/global-pull-secret-syncer-s8699","kube-system/konnectivity-agent-d2xpr","kube-system/kube-apiserver-proxy-ip-10-0-139-140.ec2.internal","openshift-cluster-node-tuning-operator/tuned-ssjlf","openshift-dns/node-resolver-mnrzt","openshift-image-registry/node-ca-hlqbr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal","openshift-multus/multus-d68vg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"]
Sep 11 16:02:22.488577 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.488555    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d2xpr"
Sep 11 16:02:22.490986 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.490942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.491566 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.491541    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 16:02:22.491566 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.491558    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 16:02:22.491689 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.491593    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xmj6x\""
Sep 11 16:02:22.493427 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.493404    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:22.493524 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:22.493489    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8j7q7" podUID="82491956-7b84-4719-b383-9dd3e987931b"
Sep 11 16:02:22.493832 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.493814    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hk6qk\""
Sep 11 16:02:22.494022 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.494006    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 16:02:22.494210 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.494195    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 16:02:22.494563 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.494545    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 16:02:22.494758 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.494743    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 16:02:22.495012 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.494993    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 16:02:22.495939 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.495902    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pdmrf"
Sep 11 16:02:22.498287 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.498268    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.498682 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.498662    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:02:22.498979 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.498960    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rz6hd\""
Sep 11 16:02:22.499340 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.499319    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 16:02:22.499430 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.499340    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 16:02:22.500755 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.500733    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 16:02:22.500835 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.500744    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:22.500879 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:22.500844    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s8699" podUID="14f2dae2-6830-49c3-8f40-786b7f33ec2d"
Sep 11 16:02:22.501194 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.501145    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 16:02:22.501620 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.501596    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 16:02:22.501705 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.501629    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 16:02:22.502652 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.502363    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 16:02:22.502652 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.502559    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 16:02:22.502811 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.502796    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5b5zx\""
Sep 11 16:02:22.503781 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.503760    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.505885 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.505858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0d91384-95ee-4acf-a00f-3becd929b46c-host-slash\") pod \"iptables-alerter-pdmrf\" (UID: \"d0d91384-95ee-4acf-a00f-3becd929b46c\") " pod="openshift-network-operator/iptables-alerter-pdmrf"
Sep 11 16:02:22.506012 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.505896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c9f7e6e-73fc-4d05-b2a3-bffad79fa6cf-agent-certs\") pod \"konnectivity-agent-d2xpr\" (UID: \"4c9f7e6e-73fc-4d05-b2a3-bffad79fa6cf\") " pod="kube-system/konnectivity-agent-d2xpr"
Sep 11 16:02:22.506012 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.505924    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-cnibin\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.506012 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.505949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-host-var-lib-cni-multus\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.506012 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.505978    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-hostroot\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.506012 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.506001    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-multus-conf-dir\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.506303 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.506029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-cni-binary-copy\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.506303 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.506055    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-host-var-lib-kubelet\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.506303 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.506100    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-host-run-multus-certs\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.506303 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.506127    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwpdk\" (UniqueName: \"kubernetes.io/projected/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-kube-api-access-bwpdk\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.506303 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.506150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lspjk\" (UniqueName: \"kubernetes.io/projected/d0d91384-95ee-4acf-a00f-3becd929b46c-kube-api-access-lspjk\") pod \"iptables-alerter-pdmrf\" (UID: \"d0d91384-95ee-4acf-a00f-3becd929b46c\") " pod="openshift-network-operator/iptables-alerter-pdmrf"
Sep 11 16:02:22.506303 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.506173    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c9f7e6e-73fc-4d05-b2a3-bffad79fa6cf-konnectivity-ca\") pod \"konnectivity-agent-d2xpr\" (UID: \"4c9f7e6e-73fc-4d05-b2a3-bffad79fa6cf\") " pod="kube-system/konnectivity-agent-d2xpr"
Sep 11 16:02:22.506303 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.506196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-system-cni-dir\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.506303 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.506222    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-multus-socket-dir-parent\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.506303 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.506249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-host-run-k8s-cni-cncf-io\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.506303 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.506273    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-host-run-netns\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.506303 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.506299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-host-var-lib-cni-bin\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.506840 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.506358    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-etc-kubernetes\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.506840 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.506465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mvdl\" (UniqueName: \"kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl\") pod \"network-check-target-8j7q7\" (UID: \"82491956-7b84-4719-b383-9dd3e987931b\") " pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:22.506840 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.506485    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d0d91384-95ee-4acf-a00f-3becd929b46c-iptables-alerter-script\") pod \"iptables-alerter-pdmrf\" (UID: \"d0d91384-95ee-4acf-a00f-3becd929b46c\") " pod="openshift-network-operator/iptables-alerter-pdmrf"
Sep 11 16:02:22.506840 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.506503    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-multus-cni-dir\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.506840 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.506518    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-multus-daemon-config\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.506840 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.506596    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.506840 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.506824    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:02:22.507216 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.507056    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 16:02:22.507216 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.507211    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6hhxw\""
Sep 11 16:02:22.508950 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.508931    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:22.509118 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.508994    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 16:02:22.509118 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:22.509033    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vw4vl" podUID="3c6a5b81-20d0-4f62-b730-356ee812b681"
Sep 11 16:02:22.509357 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.509329    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 16:02:22.509357 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.509354    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rdv9x\""
Sep 11 16:02:22.511606 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.511588    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mnrzt"
Sep 11 16:02:22.511951 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.511928    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 16:02:22.514079 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.514057    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hlqbr"
Sep 11 16:02:22.514530 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.514508    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nn4b2\""
Sep 11 16:02:22.514807 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.514790    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 16:02:22.514875 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.514861    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 16:02:22.516745 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.516420    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 16:02:22.517003 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.516863    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.517976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.517955    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-958xn\""
Sep 11 16:02:22.518101 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.517960    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 16:02:22.518101 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.517961    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 16:02:22.521297 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.521269    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 16:02:22.521398 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.521345    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 16:02:22.521398 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.521379    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mczd9\""
Sep 11 16:02:22.522043 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.522023    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 16:02:22.554726 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.554701    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-w75c2"
Sep 11 16:02:22.568343 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.568315    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-w75c2"
Sep 11 16:02:22.597231 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.597199    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 16:02:22.606799 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.606766    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/07d57c04-0697-419b-9d3e-ca11761da85c-ovnkube-config\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.606986 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.606809    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret\") pod \"global-pull-secret-syncer-s8699\" (UID: \"14f2dae2-6830-49c3-8f40-786b7f33ec2d\") " pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:22.606986 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.606838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a790ffd9-de74-4a62-9e37-d0ab20455b53-hosts-file\") pod \"node-resolver-mnrzt\" (UID: \"a790ffd9-de74-4a62-9e37-d0ab20455b53\") " pod="openshift-dns/node-resolver-mnrzt"
Sep 11 16:02:22.606986 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.606861    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12576b4c-228c-4ed2-ad49-3f8ab3a24462-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.606986 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.606890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-multus-conf-dir\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.606986 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.606914    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-systemd-units\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.606986 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.606939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86dab592-922f-41e5-bcfc-5edc18d28569-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.606986 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.606960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-multus-conf-dir\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.606986 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.606964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a790ffd9-de74-4a62-9e37-d0ab20455b53-tmp-dir\") pod \"node-resolver-mnrzt\" (UID: \"a790ffd9-de74-4a62-9e37-d0ab20455b53\") " pod="openshift-dns/node-resolver-mnrzt"
Sep 11 16:02:22.606986 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.606987    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58ea2e2b-bb7b-4537-aebf-68daed411f86-host\") pod \"node-ca-hlqbr\" (UID: \"58ea2e2b-bb7b-4537-aebf-68daed411f86\") " pod="openshift-image-registry/node-ca-hlqbr"
Sep 11 16:02:22.607382 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/07d57c04-0697-419b-9d3e-ca11761da85c-ovn-node-metrics-cert\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.607382 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607094    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-system-cni-dir\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.607382 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-multus-socket-dir-parent\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.607382 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-host-run-k8s-cni-cncf-io\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.607541 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-etc-kubernetes\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.607541 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607429    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c2w2\" (UniqueName: \"kubernetes.io/projected/5f2f3c80-6520-4b83-9b00-04f95a2167b3-kube-api-access-7c2w2\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.607541 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607451    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-etc-openvswitch\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.607541 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-run-ovn\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.607541 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-cni-bin\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.607541 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607523    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-system-cni-dir\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.607541 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mvdl\" (UniqueName: \"kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl\") pod \"network-check-target-8j7q7\" (UID: \"82491956-7b84-4719-b383-9dd3e987931b\") " pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:22.607772 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-multus-daemon-config\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.607772 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607584    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-run\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.607772 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-lib-modules\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.607772 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607625    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-kubelet\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.607772 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-run-systemd\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.607772 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607670    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmt4x\" (UniqueName: \"kubernetes.io/projected/07d57c04-0697-419b-9d3e-ca11761da85c-kube-api-access-nmt4x\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.607772 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-cnibin\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.607772 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607731    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-modprobe-d\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.607772 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607756    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-run-openvswitch\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.608069 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/86dab592-922f-41e5-bcfc-5edc18d28569-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.608069 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-cni-binary-copy\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.608069 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwpdk\" (UniqueName: \"kubernetes.io/projected/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-kube-api-access-bwpdk\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.608069 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-log-socket\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.608069 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607885    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/07d57c04-0697-419b-9d3e-ca11761da85c-env-overrides\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.608069 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607910    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lspjk\" (UniqueName: \"kubernetes.io/projected/d0d91384-95ee-4acf-a00f-3becd929b46c-kube-api-access-lspjk\") pod \"iptables-alerter-pdmrf\" (UID: \"d0d91384-95ee-4acf-a00f-3becd929b46c\") " pod="openshift-network-operator/iptables-alerter-pdmrf"
Sep 11 16:02:22.608069 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-host-run-netns\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.608069 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-systemd\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.608069 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.607981    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-host\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.608069 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.608017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12576b4c-228c-4ed2-ad49-3f8ab3a24462-etc-selinux\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.608595 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.608044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d0d91384-95ee-4acf-a00f-3becd929b46c-iptables-alerter-script\") pod \"iptables-alerter-pdmrf\" (UID: \"d0d91384-95ee-4acf-a00f-3becd929b46c\") " pod="openshift-network-operator/iptables-alerter-pdmrf"
Sep 11 16:02:22.608595 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.608423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs\") pod \"network-metrics-daemon-vw4vl\" (UID: \"3c6a5b81-20d0-4f62-b730-356ee812b681\") " pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:22.609004 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.608470    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58ea2e2b-bb7b-4537-aebf-68daed411f86-serviceca\") pod \"node-ca-hlqbr\" (UID: \"58ea2e2b-bb7b-4537-aebf-68daed411f86\") " pod="openshift-image-registry/node-ca-hlqbr"
Sep 11 16:02:22.609004 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.608971    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-slash\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.609184 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.609020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12576b4c-228c-4ed2-ad49-3f8ab3a24462-sys-fs\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.609184 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.609056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c9f7e6e-73fc-4d05-b2a3-bffad79fa6cf-agent-certs\") pod \"konnectivity-agent-d2xpr\" (UID: \"4c9f7e6e-73fc-4d05-b2a3-bffad79fa6cf\") " pod="kube-system/konnectivity-agent-d2xpr"
Sep 11 16:02:22.609184 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.609099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-hostroot\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.609345 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.609328    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-sys\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.609400 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.609367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fbjs\" (UniqueName: \"kubernetes.io/projected/3c6a5b81-20d0-4f62-b730-356ee812b681-kube-api-access-9fbjs\") pod \"network-metrics-daemon-vw4vl\" (UID: \"3c6a5b81-20d0-4f62-b730-356ee812b681\") " pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:22.609459 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.609402    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/14f2dae2-6830-49c3-8f40-786b7f33ec2d-dbus\") pod \"global-pull-secret-syncer-s8699\" (UID: \"14f2dae2-6830-49c3-8f40-786b7f33ec2d\") " pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:22.609459 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.609436    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-sysconfig\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.609629 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.609610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-kubernetes\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.609704 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.609676    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-var-lib-openvswitch\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.609760 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.609717    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/14f2dae2-6830-49c3-8f40-786b7f33ec2d-kubelet-config\") pod \"global-pull-secret-syncer-s8699\" (UID: \"14f2dae2-6830-49c3-8f40-786b7f33ec2d\") " pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:22.609810 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.609753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksqd9\" (UniqueName: \"kubernetes.io/projected/86dab592-922f-41e5-bcfc-5edc18d28569-kube-api-access-ksqd9\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.609810 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.609792    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jprw\" (UniqueName: \"kubernetes.io/projected/12576b4c-228c-4ed2-ad49-3f8ab3a24462-kube-api-access-7jprw\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.610049 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.609828    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-run-ovn-kubernetes\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.610127 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.610072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c9f7e6e-73fc-4d05-b2a3-bffad79fa6cf-konnectivity-ca\") pod \"konnectivity-agent-d2xpr\" (UID: \"4c9f7e6e-73fc-4d05-b2a3-bffad79fa6cf\") " pod="kube-system/konnectivity-agent-d2xpr"
Sep 11 16:02:22.610171 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.610124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f2f3c80-6520-4b83-9b00-04f95a2167b3-tmp\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.610171 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.610160    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86dab592-922f-41e5-bcfc-5edc18d28569-cnibin\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.610237 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.610195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/86dab592-922f-41e5-bcfc-5edc18d28569-cni-binary-copy\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.610271 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.610227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvdsz\" (UniqueName: \"kubernetes.io/projected/a790ffd9-de74-4a62-9e37-d0ab20455b53-kube-api-access-vvdsz\") pod \"node-resolver-mnrzt\" (UID: \"a790ffd9-de74-4a62-9e37-d0ab20455b53\") " pod="openshift-dns/node-resolver-mnrzt"
Sep 11 16:02:22.611250 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.611222    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-multus-socket-dir-parent\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.611342 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.611240    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12576b4c-228c-4ed2-ad49-3f8ab3a24462-device-dir\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.611342 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.611324    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-host-run-k8s-cni-cncf-io\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.611459 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.611399    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/86dab592-922f-41e5-bcfc-5edc18d28569-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.611459 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.611430    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-cnibin\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.611459 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.611443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0d91384-95ee-4acf-a00f-3becd929b46c-host-slash\") pod \"iptables-alerter-pdmrf\" (UID: \"d0d91384-95ee-4acf-a00f-3becd929b46c\") " pod="openshift-network-operator/iptables-alerter-pdmrf"
Sep 11 16:02:22.611627 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.611578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-host-var-lib-cni-multus\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.611627 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.611615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-var-lib-kubelet\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.611738 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.611651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-cni-netd\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.613024 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.612725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.613024 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.612810    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-etc-kubernetes\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.613498 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.613473    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-multus-daemon-config\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.614380 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.613935    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-cni-binary-copy\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.614380 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.614022    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0d91384-95ee-4acf-a00f-3becd929b46c-host-slash\") pod \"iptables-alerter-pdmrf\" (UID: \"d0d91384-95ee-4acf-a00f-3becd929b46c\") " pod="openshift-network-operator/iptables-alerter-pdmrf"
Sep 11 16:02:22.614380 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.614071    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-host-var-lib-cni-multus\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.614380 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.614214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/07d57c04-0697-419b-9d3e-ca11761da85c-ovnkube-script-lib\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.614380 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:22.614309    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:02:22.614380 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:22.614333    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:02:22.614380 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:22.614350    2577 projected.go:196] Error preparing data for projected volume kube-api-access-8mvdl for pod openshift-network-diagnostics/network-check-target-8j7q7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:02:22.614380 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.614378    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-host-run-netns\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.614839 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.614522    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-hostroot\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.614839 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:22.614632    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl podName:82491956-7b84-4719-b383-9dd3e987931b nodeName:}" failed. No retries permitted until 2026-09-11 16:02:23.114560313 +0000 UTC m=+2.960203864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8mvdl" (UniqueName: "kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl") pod "network-check-target-8j7q7" (UID: "82491956-7b84-4719-b383-9dd3e987931b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:02:22.614839 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.614662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86dab592-922f-41e5-bcfc-5edc18d28569-system-cni-dir\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.614839 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.614711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-host-var-lib-kubelet\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.614839 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.614751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-host-run-multus-certs\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.615124 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.614849    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-host-var-lib-kubelet\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.615124 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.614920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-tuned\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.615124 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.614996    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-host-run-multus-certs\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.615124 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.615064    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c9f7e6e-73fc-4d05-b2a3-bffad79fa6cf-konnectivity-ca\") pod \"konnectivity-agent-d2xpr\" (UID: \"4c9f7e6e-73fc-4d05-b2a3-bffad79fa6cf\") " pod="kube-system/konnectivity-agent-d2xpr"
Sep 11 16:02:22.615312 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.615122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84b2w\" (UniqueName: \"kubernetes.io/projected/58ea2e2b-bb7b-4537-aebf-68daed411f86-kube-api-access-84b2w\") pod \"node-ca-hlqbr\" (UID: \"58ea2e2b-bb7b-4537-aebf-68daed411f86\") " pod="openshift-image-registry/node-ca-hlqbr"
Sep 11 16:02:22.615312 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.615178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-host-var-lib-cni-bin\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.615312 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.615216    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-sysctl-d\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.615312 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.615253    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-host-var-lib-cni-bin\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.615312 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.615295    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-sysctl-conf\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.615544 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.615322    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-run-netns\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.615544 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.615339    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 16:02:22.615544 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.615374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-node-log\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.615544 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.615382    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d0d91384-95ee-4acf-a00f-3becd929b46c-iptables-alerter-script\") pod \"iptables-alerter-pdmrf\" (UID: \"d0d91384-95ee-4acf-a00f-3becd929b46c\") " pod="openshift-network-operator/iptables-alerter-pdmrf"
Sep 11 16:02:22.615812 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.615771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12576b4c-228c-4ed2-ad49-3f8ab3a24462-socket-dir\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.615883 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.615838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12576b4c-228c-4ed2-ad49-3f8ab3a24462-registration-dir\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.615944 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.615891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-multus-cni-dir\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.615995 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.615962    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-multus-cni-dir\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.622782 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.622716    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c9f7e6e-73fc-4d05-b2a3-bffad79fa6cf-agent-certs\") pod \"konnectivity-agent-d2xpr\" (UID: \"4c9f7e6e-73fc-4d05-b2a3-bffad79fa6cf\") " pod="kube-system/konnectivity-agent-d2xpr"
Sep 11 16:02:22.625666 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.625576    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lspjk\" (UniqueName: \"kubernetes.io/projected/d0d91384-95ee-4acf-a00f-3becd929b46c-kube-api-access-lspjk\") pod \"iptables-alerter-pdmrf\" (UID: \"d0d91384-95ee-4acf-a00f-3becd929b46c\") " pod="openshift-network-operator/iptables-alerter-pdmrf"
Sep 11 16:02:22.625666 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.625601    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwpdk\" (UniqueName: \"kubernetes.io/projected/ba0fbda3-8871-4fe6-bdae-4c744c5be5e9-kube-api-access-bwpdk\") pod \"multus-d68vg\" (UID: \"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9\") " pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.633564 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.633459    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.ec2.internal" event={"ID":"615418bae492da132dc002c225bf9501","Type":"ContainerStarted","Data":"25867f28953067922ac810b74518e91a87f3f20ff781ba3098af2dee4f535df8"}
Sep 11 16:02:22.635499 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.635436    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal" event={"ID":"0c6d1b33687c0d99c8bd06c017ca0016","Type":"ContainerStarted","Data":"a9c83277aef6ed9e8629cacabdee71f47df0da5f4d0a3c8bde90a1ae2e931453"}
Sep 11 16:02:22.716822 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.716784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-systemd\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.716822 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.716831    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-host\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.717090 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.716857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12576b4c-228c-4ed2-ad49-3f8ab3a24462-etc-selinux\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.717090 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.716884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs\") pod \"network-metrics-daemon-vw4vl\" (UID: \"3c6a5b81-20d0-4f62-b730-356ee812b681\") " pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:22.717090 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.716909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58ea2e2b-bb7b-4537-aebf-68daed411f86-serviceca\") pod \"node-ca-hlqbr\" (UID: \"58ea2e2b-bb7b-4537-aebf-68daed411f86\") " pod="openshift-image-registry/node-ca-hlqbr"
Sep 11 16:02:22.717090 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.716934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-slash\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.717090 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.716959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12576b4c-228c-4ed2-ad49-3f8ab3a24462-sys-fs\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.717090 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.716986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-sys\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.717090 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fbjs\" (UniqueName: \"kubernetes.io/projected/3c6a5b81-20d0-4f62-b730-356ee812b681-kube-api-access-9fbjs\") pod \"network-metrics-daemon-vw4vl\" (UID: \"3c6a5b81-20d0-4f62-b730-356ee812b681\") " pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:22.717090 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/14f2dae2-6830-49c3-8f40-786b7f33ec2d-dbus\") pod \"global-pull-secret-syncer-s8699\" (UID: \"14f2dae2-6830-49c3-8f40-786b7f33ec2d\") " pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:22.717090 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-sysconfig\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.717545 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-kubernetes\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.717545 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717132    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-var-lib-openvswitch\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.717545 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/14f2dae2-6830-49c3-8f40-786b7f33ec2d-kubelet-config\") pod \"global-pull-secret-syncer-s8699\" (UID: \"14f2dae2-6830-49c3-8f40-786b7f33ec2d\") " pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:22.717545 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksqd9\" (UniqueName: \"kubernetes.io/projected/86dab592-922f-41e5-bcfc-5edc18d28569-kube-api-access-ksqd9\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.717545 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jprw\" (UniqueName: \"kubernetes.io/projected/12576b4c-228c-4ed2-ad49-3f8ab3a24462-kube-api-access-7jprw\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.717545 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717255    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-run-ovn-kubernetes\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.717545 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f2f3c80-6520-4b83-9b00-04f95a2167b3-tmp\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.717545 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86dab592-922f-41e5-bcfc-5edc18d28569-cnibin\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.717545 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/86dab592-922f-41e5-bcfc-5edc18d28569-cni-binary-copy\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.717545 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvdsz\" (UniqueName: \"kubernetes.io/projected/a790ffd9-de74-4a62-9e37-d0ab20455b53-kube-api-access-vvdsz\") pod \"node-resolver-mnrzt\" (UID: \"a790ffd9-de74-4a62-9e37-d0ab20455b53\") " pod="openshift-dns/node-resolver-mnrzt"
Sep 11 16:02:22.717545 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12576b4c-228c-4ed2-ad49-3f8ab3a24462-device-dir\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.717545 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/86dab592-922f-41e5-bcfc-5edc18d28569-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.717545 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-var-lib-kubelet\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.717545 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717510    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-cni-netd\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.717545 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.718167 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/07d57c04-0697-419b-9d3e-ca11761da85c-ovnkube-script-lib\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.718167 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717601    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86dab592-922f-41e5-bcfc-5edc18d28569-system-cni-dir\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.718167 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-tuned\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.718167 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84b2w\" (UniqueName: \"kubernetes.io/projected/58ea2e2b-bb7b-4537-aebf-68daed411f86-kube-api-access-84b2w\") pod \"node-ca-hlqbr\" (UID: \"58ea2e2b-bb7b-4537-aebf-68daed411f86\") " pod="openshift-image-registry/node-ca-hlqbr"
Sep 11 16:02:22.718167 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717687    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-sysctl-d\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.718167 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-sysctl-conf\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.718167 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717740    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-run-netns\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.718167 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717774    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-node-log\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.718167 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12576b4c-228c-4ed2-ad49-3f8ab3a24462-socket-dir\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.718167 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12576b4c-228c-4ed2-ad49-3f8ab3a24462-registration-dir\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.718167 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/07d57c04-0697-419b-9d3e-ca11761da85c-ovnkube-config\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.718167 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret\") pod \"global-pull-secret-syncer-s8699\" (UID: \"14f2dae2-6830-49c3-8f40-786b7f33ec2d\") " pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:22.718167 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a790ffd9-de74-4a62-9e37-d0ab20455b53-hosts-file\") pod \"node-resolver-mnrzt\" (UID: \"a790ffd9-de74-4a62-9e37-d0ab20455b53\") " pod="openshift-dns/node-resolver-mnrzt"
Sep 11 16:02:22.718167 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12576b4c-228c-4ed2-ad49-3f8ab3a24462-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.718167 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-systemd-units\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.718167 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717933    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86dab592-922f-41e5-bcfc-5edc18d28569-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.718167 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a790ffd9-de74-4a62-9e37-d0ab20455b53-tmp-dir\") pod \"node-resolver-mnrzt\" (UID: \"a790ffd9-de74-4a62-9e37-d0ab20455b53\") " pod="openshift-dns/node-resolver-mnrzt"
Sep 11 16:02:22.719001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58ea2e2b-bb7b-4537-aebf-68daed411f86-host\") pod \"node-ca-hlqbr\" (UID: \"58ea2e2b-bb7b-4537-aebf-68daed411f86\") " pod="openshift-image-registry/node-ca-hlqbr"
Sep 11 16:02:22.719001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.717980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/07d57c04-0697-419b-9d3e-ca11761da85c-ovn-node-metrics-cert\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.719001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718003    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c2w2\" (UniqueName: \"kubernetes.io/projected/5f2f3c80-6520-4b83-9b00-04f95a2167b3-kube-api-access-7c2w2\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.719001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718019    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-etc-openvswitch\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.719001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-run-ovn\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.719001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-cni-bin\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.719001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-run\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.719001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718155    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-kubernetes\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.719001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718163    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-cni-netd\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.719001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718162    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/14f2dae2-6830-49c3-8f40-786b7f33ec2d-kubelet-config\") pod \"global-pull-secret-syncer-s8699\" (UID: \"14f2dae2-6830-49c3-8f40-786b7f33ec2d\") " pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:22.719001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-slash\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.719001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-lib-modules\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.719001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718219    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-sysconfig\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.719001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-kubelet\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.719001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-run-systemd\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.719001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718251    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-var-lib-kubelet\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.719001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmt4x\" (UniqueName: \"kubernetes.io/projected/07d57c04-0697-419b-9d3e-ca11761da85c-kube-api-access-nmt4x\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.719001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-modprobe-d\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.720001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718281    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.720001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-run-openvswitch\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.720001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718301    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-lib-modules\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.720001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718307    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-var-lib-openvswitch\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.720001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/86dab592-922f-41e5-bcfc-5edc18d28569-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.720001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718347    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/12576b4c-228c-4ed2-ad49-3f8ab3a24462-sys-fs\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.720001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718355    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-run-openvswitch\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.720001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-log-socket\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.720001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/07d57c04-0697-419b-9d3e-ca11761da85c-env-overrides\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.720001 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:22.718413    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:02:22.720001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718441    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/12576b4c-228c-4ed2-ad49-3f8ab3a24462-etc-selinux\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.720001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718446    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-sys\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.720001 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:22.718471    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret podName:14f2dae2-6830-49c3-8f40-786b7f33ec2d nodeName:}" failed. No retries permitted until 2026-09-11 16:02:23.218457447 +0000 UTC m=+3.064100982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret") pod "global-pull-secret-syncer-s8699" (UID: "14f2dae2-6830-49c3-8f40-786b7f33ec2d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:02:22.720001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718497    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86dab592-922f-41e5-bcfc-5edc18d28569-system-cni-dir\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.720001 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:22.718502    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:02:22.720001 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:22.718550    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs podName:3c6a5b81-20d0-4f62-b730-356ee812b681 nodeName:}" failed. No retries permitted until 2026-09-11 16:02:23.218536475 +0000 UTC m=+3.064180009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs") pod "network-metrics-daemon-vw4vl" (UID: "3c6a5b81-20d0-4f62-b730-356ee812b681") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:02:22.720944 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718711    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-run-ovn-kubernetes\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.720944 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718849    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/07d57c04-0697-419b-9d3e-ca11761da85c-ovnkube-script-lib\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.720944 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718887    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58ea2e2b-bb7b-4537-aebf-68daed411f86-serviceca\") pod \"node-ca-hlqbr\" (UID: \"58ea2e2b-bb7b-4537-aebf-68daed411f86\") " pod="openshift-image-registry/node-ca-hlqbr"
Sep 11 16:02:22.720944 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718909    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/07d57c04-0697-419b-9d3e-ca11761da85c-env-overrides\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.720944 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718923    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-cni-bin\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.720944 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718928    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a790ffd9-de74-4a62-9e37-d0ab20455b53-hosts-file\") pod \"node-resolver-mnrzt\" (UID: \"a790ffd9-de74-4a62-9e37-d0ab20455b53\") " pod="openshift-dns/node-resolver-mnrzt"
Sep 11 16:02:22.720944 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718949    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-kubelet\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.720944 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718973    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-run-systemd\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.720944 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718998    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58ea2e2b-bb7b-4537-aebf-68daed411f86-host\") pod \"node-ca-hlqbr\" (UID: \"58ea2e2b-bb7b-4537-aebf-68daed411f86\") " pod="openshift-image-registry/node-ca-hlqbr"
Sep 11 16:02:22.720944 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718997    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/12576b4c-228c-4ed2-ad49-3f8ab3a24462-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.720944 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.719099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-modprobe-d\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.720944 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.719251    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-etc-openvswitch\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.720944 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.719302    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-run-ovn\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.720944 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.719533    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86dab592-922f-41e5-bcfc-5edc18d28569-cnibin\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.720944 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.719549    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a790ffd9-de74-4a62-9e37-d0ab20455b53-tmp-dir\") pod \"node-resolver-mnrzt\" (UID: \"a790ffd9-de74-4a62-9e37-d0ab20455b53\") " pod="openshift-dns/node-resolver-mnrzt"
Sep 11 16:02:22.720944 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.719626    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-systemd-units\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.720944 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.719830    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/14f2dae2-6830-49c3-8f40-786b7f33ec2d-dbus\") pod \"global-pull-secret-syncer-s8699\" (UID: \"14f2dae2-6830-49c3-8f40-786b7f33ec2d\") " pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:22.720944 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.719917    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/12576b4c-228c-4ed2-ad49-3f8ab3a24462-device-dir\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.721996 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.719974    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-run\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.721996 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.718336    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-systemd\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.721996 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.719993    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86dab592-922f-41e5-bcfc-5edc18d28569-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.721996 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.720138    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/86dab592-922f-41e5-bcfc-5edc18d28569-cni-binary-copy\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.721996 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.720143    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-sysctl-d\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.721996 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.720177    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-sysctl-conf\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.721996 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.720185    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-host-run-netns\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.721996 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.720234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f2f3c80-6520-4b83-9b00-04f95a2167b3-host\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.721996 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.720308    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-node-log\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.721996 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.720415    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/12576b4c-228c-4ed2-ad49-3f8ab3a24462-socket-dir\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.721996 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.720423    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/12576b4c-228c-4ed2-ad49-3f8ab3a24462-registration-dir\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.721996 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.720469    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/07d57c04-0697-419b-9d3e-ca11761da85c-log-socket\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.721996 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.720526    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/86dab592-922f-41e5-bcfc-5edc18d28569-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.721996 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.720623    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/86dab592-922f-41e5-bcfc-5edc18d28569-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.721996 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.720950    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/07d57c04-0697-419b-9d3e-ca11761da85c-ovnkube-config\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.721996 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.721662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f2f3c80-6520-4b83-9b00-04f95a2167b3-etc-tuned\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.721996 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.721837    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f2f3c80-6520-4b83-9b00-04f95a2167b3-tmp\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.723190 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.723141    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/07d57c04-0697-419b-9d3e-ca11761da85c-ovn-node-metrics-cert\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.734255 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.734128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fbjs\" (UniqueName: \"kubernetes.io/projected/3c6a5b81-20d0-4f62-b730-356ee812b681-kube-api-access-9fbjs\") pod \"network-metrics-daemon-vw4vl\" (UID: \"3c6a5b81-20d0-4f62-b730-356ee812b681\") " pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:22.735211 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.735186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksqd9\" (UniqueName: \"kubernetes.io/projected/86dab592-922f-41e5-bcfc-5edc18d28569-kube-api-access-ksqd9\") pod \"multus-additional-cni-plugins-dn4s5\" (UID: \"86dab592-922f-41e5-bcfc-5edc18d28569\") " pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.735343 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.735254    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvdsz\" (UniqueName: \"kubernetes.io/projected/a790ffd9-de74-4a62-9e37-d0ab20455b53-kube-api-access-vvdsz\") pod \"node-resolver-mnrzt\" (UID: \"a790ffd9-de74-4a62-9e37-d0ab20455b53\") " pod="openshift-dns/node-resolver-mnrzt"
Sep 11 16:02:22.735786 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.735703    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmt4x\" (UniqueName: \"kubernetes.io/projected/07d57c04-0697-419b-9d3e-ca11761da85c-kube-api-access-nmt4x\") pod \"ovnkube-node-ng2ww\" (UID: \"07d57c04-0697-419b-9d3e-ca11761da85c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.736613 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.736569    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c2w2\" (UniqueName: \"kubernetes.io/projected/5f2f3c80-6520-4b83-9b00-04f95a2167b3-kube-api-access-7c2w2\") pod \"tuned-ssjlf\" (UID: \"5f2f3c80-6520-4b83-9b00-04f95a2167b3\") " pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.737419 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.737398    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jprw\" (UniqueName: \"kubernetes.io/projected/12576b4c-228c-4ed2-ad49-3f8ab3a24462-kube-api-access-7jprw\") pod \"aws-ebs-csi-driver-node-hdgnr\" (UID: \"12576b4c-228c-4ed2-ad49-3f8ab3a24462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.738121 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.738100    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84b2w\" (UniqueName: \"kubernetes.io/projected/58ea2e2b-bb7b-4537-aebf-68daed411f86-kube-api-access-84b2w\") pod \"node-ca-hlqbr\" (UID: \"58ea2e2b-bb7b-4537-aebf-68daed411f86\") " pod="openshift-image-registry/node-ca-hlqbr"
Sep 11 16:02:22.800934 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.800895    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d2xpr"
Sep 11 16:02:22.811791 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.811596    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d68vg"
Sep 11 16:02:22.823054 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.823014    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pdmrf"
Sep 11 16:02:22.829561 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.829350    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mnrzt"
Sep 11 16:02:22.836967 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.836757    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:22.845117 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.845084    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ssjlf"
Sep 11 16:02:22.850682 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.850638    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dn4s5"
Sep 11 16:02:22.859899 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.859868    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hlqbr"
Sep 11 16:02:22.867209 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.867177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr"
Sep 11 16:02:22.912849 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:22.912820    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:02:23.122665 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:23.122621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mvdl\" (UniqueName: \"kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl\") pod \"network-check-target-8j7q7\" (UID: \"82491956-7b84-4719-b383-9dd3e987931b\") " pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:23.122848 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:23.122797    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:02:23.122848 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:23.122836    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:02:23.122972 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:23.122851    2577 projected.go:196] Error preparing data for projected volume kube-api-access-8mvdl for pod openshift-network-diagnostics/network-check-target-8j7q7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:02:23.122972 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:23.122918    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl podName:82491956-7b84-4719-b383-9dd3e987931b nodeName:}" failed. No retries permitted until 2026-09-11 16:02:24.122899749 +0000 UTC m=+3.968543287 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mvdl" (UniqueName: "kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl") pod "network-check-target-8j7q7" (UID: "82491956-7b84-4719-b383-9dd3e987931b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:02:23.223311 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:23.223273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret\") pod \"global-pull-secret-syncer-s8699\" (UID: \"14f2dae2-6830-49c3-8f40-786b7f33ec2d\") " pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:23.223508 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:23.223348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs\") pod \"network-metrics-daemon-vw4vl\" (UID: \"3c6a5b81-20d0-4f62-b730-356ee812b681\") " pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:23.223508 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:23.223461    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:02:23.223508 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:23.223461    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:02:23.223653 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:23.223522    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs podName:3c6a5b81-20d0-4f62-b730-356ee812b681 nodeName:}" failed. No retries permitted until 2026-09-11 16:02:24.223501568 +0000 UTC m=+4.069145118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs") pod "network-metrics-daemon-vw4vl" (UID: "3c6a5b81-20d0-4f62-b730-356ee812b681") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:02:23.223653 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:23.223540    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret podName:14f2dae2-6830-49c3-8f40-786b7f33ec2d nodeName:}" failed. No retries permitted until 2026-09-11 16:02:24.22353062 +0000 UTC m=+4.069174156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret") pod "global-pull-secret-syncer-s8699" (UID: "14f2dae2-6830-49c3-8f40-786b7f33ec2d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:02:23.465905 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:23.465815    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:02:23.572392 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:23.572166    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 15:57:22 +0000 UTC" deadline="2028-03-18 12:00:20.951452454 +0000 UTC"
Sep 11 16:02:23.572392 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:23.572389    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13291h57m57.3790693s"
Sep 11 16:02:23.625903 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:23.625868    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:23.626116 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:23.626002    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vw4vl" podUID="3c6a5b81-20d0-4f62-b730-356ee812b681"
Sep 11 16:02:23.913265 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:23.913232    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d57c04_0697_419b_9d3e_ca11761da85c.slice/crio-a05140661b526e192ce9592702e8a7434afb7afe3a5f8601a11c8318a13c5c6b WatchSource:0}: Error finding container a05140661b526e192ce9592702e8a7434afb7afe3a5f8601a11c8318a13c5c6b: Status 404 returned error can't find the container with id a05140661b526e192ce9592702e8a7434afb7afe3a5f8601a11c8318a13c5c6b
Sep 11 16:02:23.921460 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:23.921431    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86dab592_922f_41e5_bcfc_5edc18d28569.slice/crio-df7939278d3257ef95164984ea90df9b4e709fb861fa01bf5b8be68932046216 WatchSource:0}: Error finding container df7939278d3257ef95164984ea90df9b4e709fb861fa01bf5b8be68932046216: Status 404 returned error can't find the container with id df7939278d3257ef95164984ea90df9b4e709fb861fa01bf5b8be68932046216
Sep 11 16:02:23.921732 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:23.921675    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0fbda3_8871_4fe6_bdae_4c744c5be5e9.slice/crio-ef7a1d2ec490d5c6a3ff5ccee6788268d1b7221d2624dcd2e2d71fad3493b2a7 WatchSource:0}: Error finding container ef7a1d2ec490d5c6a3ff5ccee6788268d1b7221d2624dcd2e2d71fad3493b2a7: Status 404 returned error can't find the container with id ef7a1d2ec490d5c6a3ff5ccee6788268d1b7221d2624dcd2e2d71fad3493b2a7
Sep 11 16:02:23.922198 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:23.922173    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ea2e2b_bb7b_4537_aebf_68daed411f86.slice/crio-26a3e9cdee44c296f1c99cf863ef82b17570badf57238aa0c2a08157f02847c5 WatchSource:0}: Error finding container 26a3e9cdee44c296f1c99cf863ef82b17570badf57238aa0c2a08157f02847c5: Status 404 returned error can't find the container with id 26a3e9cdee44c296f1c99cf863ef82b17570badf57238aa0c2a08157f02847c5
Sep 11 16:02:23.922831 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:23.922809    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d91384_95ee_4acf_a00f_3becd929b46c.slice/crio-36b33bf8184e7d725244215da9e7003b17ccb6d84c64cb0e7d5c04f6af86a321 WatchSource:0}: Error finding container 36b33bf8184e7d725244215da9e7003b17ccb6d84c64cb0e7d5c04f6af86a321: Status 404 returned error can't find the container with id 36b33bf8184e7d725244215da9e7003b17ccb6d84c64cb0e7d5c04f6af86a321
Sep 11 16:02:23.923584 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:23.923488    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2f3c80_6520_4b83_9b00_04f95a2167b3.slice/crio-4966afd2b54f4539a163fc76efa6e4741ac3a8917aa8687e6567e0e6416dbc5c WatchSource:0}: Error finding container 4966afd2b54f4539a163fc76efa6e4741ac3a8917aa8687e6567e0e6416dbc5c: Status 404 returned error can't find the container with id 4966afd2b54f4539a163fc76efa6e4741ac3a8917aa8687e6567e0e6416dbc5c
Sep 11 16:02:23.947650 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:23.947534    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12576b4c_228c_4ed2_ad49_3f8ab3a24462.slice/crio-16e024c74d6342af397186d346e805eb71d5fa4dddbfd6fd89e32fad01b0b56a WatchSource:0}: Error finding container 16e024c74d6342af397186d346e805eb71d5fa4dddbfd6fd89e32fad01b0b56a: Status 404 returned error can't find the container with id 16e024c74d6342af397186d346e805eb71d5fa4dddbfd6fd89e32fad01b0b56a
Sep 11 16:02:23.947881 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:23.947824    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9f7e6e_73fc_4d05_b2a3_bffad79fa6cf.slice/crio-f18c926485fc20518dbb0b6288312b9648072e91d2c5c33b53d668500fbf9195 WatchSource:0}: Error finding container f18c926485fc20518dbb0b6288312b9648072e91d2c5c33b53d668500fbf9195: Status 404 returned error can't find the container with id f18c926485fc20518dbb0b6288312b9648072e91d2c5c33b53d668500fbf9195
Sep 11 16:02:23.948770 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:23.948747    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda790ffd9_de74_4a62_9e37_d0ab20455b53.slice/crio-3e95109937e0413f67a2103cb3b6445aed7be11a5d81ad2a4b05872bd139c079 WatchSource:0}: Error finding container 3e95109937e0413f67a2103cb3b6445aed7be11a5d81ad2a4b05872bd139c079: Status 404 returned error can't find the container with id 3e95109937e0413f67a2103cb3b6445aed7be11a5d81ad2a4b05872bd139c079
Sep 11 16:02:24.131893 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:24.131854    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mvdl\" (UniqueName: \"kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl\") pod \"network-check-target-8j7q7\" (UID: \"82491956-7b84-4719-b383-9dd3e987931b\") " pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:24.132072 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:24.132027    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:02:24.132072 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:24.132051    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:02:24.132072 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:24.132064    2577 projected.go:196] Error preparing data for projected volume kube-api-access-8mvdl for pod openshift-network-diagnostics/network-check-target-8j7q7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:02:24.132225 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:24.132161    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl podName:82491956-7b84-4719-b383-9dd3e987931b nodeName:}" failed. No retries permitted until 2026-09-11 16:02:26.132137964 +0000 UTC m=+5.977781518 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mvdl" (UniqueName: "kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl") pod "network-check-target-8j7q7" (UID: "82491956-7b84-4719-b383-9dd3e987931b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:02:24.233235 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:24.233201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs\") pod \"network-metrics-daemon-vw4vl\" (UID: \"3c6a5b81-20d0-4f62-b730-356ee812b681\") " pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:24.233381 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:24.233257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret\") pod \"global-pull-secret-syncer-s8699\" (UID: \"14f2dae2-6830-49c3-8f40-786b7f33ec2d\") " pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:24.233381 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:24.233361    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:02:24.233381 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:24.233371    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:02:24.233522 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:24.233429    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret podName:14f2dae2-6830-49c3-8f40-786b7f33ec2d nodeName:}" failed. No retries permitted until 2026-09-11 16:02:26.233410346 +0000 UTC m=+6.079053898 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret") pod "global-pull-secret-syncer-s8699" (UID: "14f2dae2-6830-49c3-8f40-786b7f33ec2d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:02:24.233522 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:24.233445    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs podName:3c6a5b81-20d0-4f62-b730-356ee812b681 nodeName:}" failed. No retries permitted until 2026-09-11 16:02:26.233437497 +0000 UTC m=+6.079081032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs") pod "network-metrics-daemon-vw4vl" (UID: "3c6a5b81-20d0-4f62-b730-356ee812b681") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:02:24.573645 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:24.573427    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 15:57:22 +0000 UTC" deadline="2028-02-22 23:18:25.344371734 +0000 UTC"
Sep 11 16:02:24.573645 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:24.573472    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12703h16m0.770904923s"
Sep 11 16:02:24.631494 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:24.631462    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:24.631672 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:24.631595    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s8699" podUID="14f2dae2-6830-49c3-8f40-786b7f33ec2d"
Sep 11 16:02:24.632054 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:24.632015    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:24.632166 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:24.632141    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8j7q7" podUID="82491956-7b84-4719-b383-9dd3e987931b"
Sep 11 16:02:24.663347 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:24.663302    2577 generic.go:356] "Generic (PLEG): container finished" podID="0c6d1b33687c0d99c8bd06c017ca0016" containerID="5529eafb4fac3edd01c74afdae9baa50d2ba7569a064cd3f1db8df829bc18e63" exitCode=0
Sep 11 16:02:24.663535 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:24.663402    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal" event={"ID":"0c6d1b33687c0d99c8bd06c017ca0016","Type":"ContainerDied","Data":"5529eafb4fac3edd01c74afdae9baa50d2ba7569a064cd3f1db8df829bc18e63"}
Sep 11 16:02:24.670359 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:24.670324    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dn4s5" event={"ID":"86dab592-922f-41e5-bcfc-5edc18d28569","Type":"ContainerStarted","Data":"df7939278d3257ef95164984ea90df9b4e709fb861fa01bf5b8be68932046216"}
Sep 11 16:02:24.678445 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:24.678407    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ssjlf" event={"ID":"5f2f3c80-6520-4b83-9b00-04f95a2167b3","Type":"ContainerStarted","Data":"4966afd2b54f4539a163fc76efa6e4741ac3a8917aa8687e6567e0e6416dbc5c"}
Sep 11 16:02:24.681358 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:24.681328    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pdmrf" event={"ID":"d0d91384-95ee-4acf-a00f-3becd929b46c","Type":"ContainerStarted","Data":"36b33bf8184e7d725244215da9e7003b17ccb6d84c64cb0e7d5c04f6af86a321"}
Sep 11 16:02:24.689967 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:24.689935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d68vg" event={"ID":"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9","Type":"ContainerStarted","Data":"ef7a1d2ec490d5c6a3ff5ccee6788268d1b7221d2624dcd2e2d71fad3493b2a7"}
Sep 11 16:02:24.695645 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:24.695615    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hlqbr" event={"ID":"58ea2e2b-bb7b-4537-aebf-68daed411f86","Type":"ContainerStarted","Data":"26a3e9cdee44c296f1c99cf863ef82b17570badf57238aa0c2a08157f02847c5"}
Sep 11 16:02:24.708485 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:24.708450    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww" event={"ID":"07d57c04-0697-419b-9d3e-ca11761da85c","Type":"ContainerStarted","Data":"a05140661b526e192ce9592702e8a7434afb7afe3a5f8601a11c8318a13c5c6b"}
Sep 11 16:02:24.710787 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:24.710754    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.ec2.internal" event={"ID":"615418bae492da132dc002c225bf9501","Type":"ContainerStarted","Data":"8c0e816262be9515d43ed61090a28ca21f4d5ba8b126739bb5b12a554cda080f"}
Sep 11 16:02:24.722480 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:24.722438    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mnrzt" event={"ID":"a790ffd9-de74-4a62-9e37-d0ab20455b53","Type":"ContainerStarted","Data":"3e95109937e0413f67a2103cb3b6445aed7be11a5d81ad2a4b05872bd139c079"}
Sep 11 16:02:24.724707 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:24.724673    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d2xpr" event={"ID":"4c9f7e6e-73fc-4d05-b2a3-bffad79fa6cf","Type":"ContainerStarted","Data":"f18c926485fc20518dbb0b6288312b9648072e91d2c5c33b53d668500fbf9195"}
Sep 11 16:02:24.726826 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:24.726780    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr" event={"ID":"12576b4c-228c-4ed2-ad49-3f8ab3a24462","Type":"ContainerStarted","Data":"16e024c74d6342af397186d346e805eb71d5fa4dddbfd6fd89e32fad01b0b56a"}
Sep 11 16:02:25.626689 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:25.626625    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:25.627279 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:25.626783    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vw4vl" podUID="3c6a5b81-20d0-4f62-b730-356ee812b681"
Sep 11 16:02:25.738623 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:25.738555    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal" event={"ID":"0c6d1b33687c0d99c8bd06c017ca0016","Type":"ContainerStarted","Data":"969aac10b385d3e984f82ee0e578d1a422e948132d9b02783cb2e5b5194aa3bc"}
Sep 11 16:02:25.754748 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:25.754664    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.ec2.internal" podStartSLOduration=3.7546403919999998 podStartE2EDuration="3.754640392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:02:24.735838342 +0000 UTC m=+4.581481903" watchObservedRunningTime="2026-09-11 16:02:25.754640392 +0000 UTC m=+5.600283952"
Sep 11 16:02:26.151538 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:26.151496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mvdl\" (UniqueName: \"kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl\") pod \"network-check-target-8j7q7\" (UID: \"82491956-7b84-4719-b383-9dd3e987931b\") " pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:26.151756 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:26.151706    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:02:26.151756 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:26.151727    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:02:26.151756 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:26.151742    2577 projected.go:196] Error preparing data for projected volume kube-api-access-8mvdl for pod openshift-network-diagnostics/network-check-target-8j7q7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:02:26.151930 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:26.151815    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl podName:82491956-7b84-4719-b383-9dd3e987931b nodeName:}" failed. No retries permitted until 2026-09-11 16:02:30.151784995 +0000 UTC m=+9.997428535 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mvdl" (UniqueName: "kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl") pod "network-check-target-8j7q7" (UID: "82491956-7b84-4719-b383-9dd3e987931b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:02:26.252592 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:26.252546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret\") pod \"global-pull-secret-syncer-s8699\" (UID: \"14f2dae2-6830-49c3-8f40-786b7f33ec2d\") " pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:26.252806 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:26.252629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs\") pod \"network-metrics-daemon-vw4vl\" (UID: \"3c6a5b81-20d0-4f62-b730-356ee812b681\") " pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:26.252806 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:26.252753    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:02:26.252922 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:26.252821    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs podName:3c6a5b81-20d0-4f62-b730-356ee812b681 nodeName:}" failed. No retries permitted until 2026-09-11 16:02:30.252798883 +0000 UTC m=+10.098442418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs") pod "network-metrics-daemon-vw4vl" (UID: "3c6a5b81-20d0-4f62-b730-356ee812b681") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:02:26.253272 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:26.253255    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:02:26.253352 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:26.253314    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret podName:14f2dae2-6830-49c3-8f40-786b7f33ec2d nodeName:}" failed. No retries permitted until 2026-09-11 16:02:30.25329762 +0000 UTC m=+10.098941162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret") pod "global-pull-secret-syncer-s8699" (UID: "14f2dae2-6830-49c3-8f40-786b7f33ec2d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:02:26.626892 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:26.626857    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:26.627428 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:26.626999    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s8699" podUID="14f2dae2-6830-49c3-8f40-786b7f33ec2d"
Sep 11 16:02:26.627495 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:26.627466    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:26.630194 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:26.627562    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8j7q7" podUID="82491956-7b84-4719-b383-9dd3e987931b"
Sep 11 16:02:27.626484 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:27.626446    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:27.626704 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:27.626613    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vw4vl" podUID="3c6a5b81-20d0-4f62-b730-356ee812b681"
Sep 11 16:02:28.632217 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:28.631599    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:28.632217 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:28.631721    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s8699" podUID="14f2dae2-6830-49c3-8f40-786b7f33ec2d"
Sep 11 16:02:28.632217 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:28.632146    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:28.632770 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:28.632262    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vw4vl" podUID="3c6a5b81-20d0-4f62-b730-356ee812b681"
Sep 11 16:02:28.632770 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:28.632298    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:28.632770 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:28.632341    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8j7q7" podUID="82491956-7b84-4719-b383-9dd3e987931b"
Sep 11 16:02:30.190582 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:30.190476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mvdl\" (UniqueName: \"kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl\") pod \"network-check-target-8j7q7\" (UID: \"82491956-7b84-4719-b383-9dd3e987931b\") " pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:30.191219 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:30.190620    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:02:30.191219 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:30.190640    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:02:30.191219 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:30.190652    2577 projected.go:196] Error preparing data for projected volume kube-api-access-8mvdl for pod openshift-network-diagnostics/network-check-target-8j7q7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:02:30.191219 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:30.190720    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl podName:82491956-7b84-4719-b383-9dd3e987931b nodeName:}" failed. No retries permitted until 2026-09-11 16:02:38.190701426 +0000 UTC m=+18.036344960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mvdl" (UniqueName: "kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl") pod "network-check-target-8j7q7" (UID: "82491956-7b84-4719-b383-9dd3e987931b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:02:30.291598 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:30.291554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs\") pod \"network-metrics-daemon-vw4vl\" (UID: \"3c6a5b81-20d0-4f62-b730-356ee812b681\") " pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:30.291786 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:30.291627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret\") pod \"global-pull-secret-syncer-s8699\" (UID: \"14f2dae2-6830-49c3-8f40-786b7f33ec2d\") " pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:30.291786 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:30.291741    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:02:30.291900 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:30.291798    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret podName:14f2dae2-6830-49c3-8f40-786b7f33ec2d nodeName:}" failed. No retries permitted until 2026-09-11 16:02:38.291779019 +0000 UTC m=+18.137422570 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret") pod "global-pull-secret-syncer-s8699" (UID: "14f2dae2-6830-49c3-8f40-786b7f33ec2d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:02:30.292282 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:30.292187    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:02:30.292282 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:30.292246    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs podName:3c6a5b81-20d0-4f62-b730-356ee812b681 nodeName:}" failed. No retries permitted until 2026-09-11 16:02:38.292231586 +0000 UTC m=+18.137875118 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs") pod "network-metrics-daemon-vw4vl" (UID: "3c6a5b81-20d0-4f62-b730-356ee812b681") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:02:30.630108 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:30.630073    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:30.630270 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:30.630207    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8j7q7" podUID="82491956-7b84-4719-b383-9dd3e987931b"
Sep 11 16:02:30.630626 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:30.630602    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:30.630740 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:30.630699    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s8699" podUID="14f2dae2-6830-49c3-8f40-786b7f33ec2d"
Sep 11 16:02:30.630845 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:30.630755    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:30.630845 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:30.630831    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vw4vl" podUID="3c6a5b81-20d0-4f62-b730-356ee812b681"
Sep 11 16:02:32.627179 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:32.627146    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:32.627665 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:32.627197    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:32.627665 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:32.627269    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s8699" podUID="14f2dae2-6830-49c3-8f40-786b7f33ec2d"
Sep 11 16:02:32.627665 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:32.627283    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:32.627665 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:32.627383    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vw4vl" podUID="3c6a5b81-20d0-4f62-b730-356ee812b681"
Sep 11 16:02:32.627665 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:32.627447    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8j7q7" podUID="82491956-7b84-4719-b383-9dd3e987931b"
Sep 11 16:02:34.626259 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:34.625991    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:34.626734 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:34.625991    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:34.626734 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:34.625997    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:34.626734 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:34.626383    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vw4vl" podUID="3c6a5b81-20d0-4f62-b730-356ee812b681"
Sep 11 16:02:34.626734 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:34.626474    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s8699" podUID="14f2dae2-6830-49c3-8f40-786b7f33ec2d"
Sep 11 16:02:34.626734 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:34.626573    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8j7q7" podUID="82491956-7b84-4719-b383-9dd3e987931b"
Sep 11 16:02:36.626436 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:36.626396    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:36.626436 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:36.626416    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:36.626949 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:36.626396    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:36.626949 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:36.626530    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vw4vl" podUID="3c6a5b81-20d0-4f62-b730-356ee812b681"
Sep 11 16:02:36.626949 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:36.626602    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8j7q7" podUID="82491956-7b84-4719-b383-9dd3e987931b"
Sep 11 16:02:36.626949 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:36.626677    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s8699" podUID="14f2dae2-6830-49c3-8f40-786b7f33ec2d"
Sep 11 16:02:38.250761 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:38.250722    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mvdl\" (UniqueName: \"kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl\") pod \"network-check-target-8j7q7\" (UID: \"82491956-7b84-4719-b383-9dd3e987931b\") " pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:38.251251 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:38.250837    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:02:38.251251 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:38.250854    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:02:38.251251 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:38.250863    2577 projected.go:196] Error preparing data for projected volume kube-api-access-8mvdl for pod openshift-network-diagnostics/network-check-target-8j7q7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:02:38.251251 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:38.250910    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl podName:82491956-7b84-4719-b383-9dd3e987931b nodeName:}" failed. No retries permitted until 2026-09-11 16:02:54.250896776 +0000 UTC m=+34.096540309 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mvdl" (UniqueName: "kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl") pod "network-check-target-8j7q7" (UID: "82491956-7b84-4719-b383-9dd3e987931b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:02:38.351183 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:38.351153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret\") pod \"global-pull-secret-syncer-s8699\" (UID: \"14f2dae2-6830-49c3-8f40-786b7f33ec2d\") " pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:38.351362 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:38.351217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs\") pod \"network-metrics-daemon-vw4vl\" (UID: \"3c6a5b81-20d0-4f62-b730-356ee812b681\") " pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:38.351362 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:38.351334    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:02:38.351362 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:38.351336    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:02:38.351532 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:38.351395    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs podName:3c6a5b81-20d0-4f62-b730-356ee812b681 nodeName:}" failed. No retries permitted until 2026-09-11 16:02:54.3513755 +0000 UTC m=+34.197019051 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs") pod "network-metrics-daemon-vw4vl" (UID: "3c6a5b81-20d0-4f62-b730-356ee812b681") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:02:38.351532 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:38.351413    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret podName:14f2dae2-6830-49c3-8f40-786b7f33ec2d nodeName:}" failed. No retries permitted until 2026-09-11 16:02:54.351404115 +0000 UTC m=+34.197047656 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret") pod "global-pull-secret-syncer-s8699" (UID: "14f2dae2-6830-49c3-8f40-786b7f33ec2d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:02:38.626699 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:38.626662    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:38.626903 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:38.626706    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:38.626903 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:38.626662    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:38.626903 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:38.626805    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vw4vl" podUID="3c6a5b81-20d0-4f62-b730-356ee812b681"
Sep 11 16:02:38.626903 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:38.626891    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8j7q7" podUID="82491956-7b84-4719-b383-9dd3e987931b"
Sep 11 16:02:38.627146 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:38.626964    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s8699" podUID="14f2dae2-6830-49c3-8f40-786b7f33ec2d"
Sep 11 16:02:40.626989 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:40.626948    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:40.627444 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:40.627090    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:40.627444 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:40.627112    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:40.627444 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:40.627093    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vw4vl" podUID="3c6a5b81-20d0-4f62-b730-356ee812b681"
Sep 11 16:02:40.627444 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:40.627198    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s8699" podUID="14f2dae2-6830-49c3-8f40-786b7f33ec2d"
Sep 11 16:02:40.627444 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:40.627270    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8j7q7" podUID="82491956-7b84-4719-b383-9dd3e987931b"
Sep 11 16:02:42.626889 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.626718    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:42.627420 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:42.626931    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8j7q7" podUID="82491956-7b84-4719-b383-9dd3e987931b"
Sep 11 16:02:42.627420 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.626783    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:42.627420 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:42.627007    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s8699" podUID="14f2dae2-6830-49c3-8f40-786b7f33ec2d"
Sep 11 16:02:42.627420 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.626721    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:42.627420 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:42.627135    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vw4vl" podUID="3c6a5b81-20d0-4f62-b730-356ee812b681"
Sep 11 16:02:42.773422 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.773388    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d68vg" event={"ID":"ba0fbda3-8871-4fe6-bdae-4c744c5be5e9","Type":"ContainerStarted","Data":"82db0158ab78cfcd5c8107200b7bf0a789229f2906204a8d8abe0f1d4844ec97"}
Sep 11 16:02:42.775089 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.775062    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hlqbr" event={"ID":"58ea2e2b-bb7b-4537-aebf-68daed411f86","Type":"ContainerStarted","Data":"12eeef88f9cef90771e131210cf86e0009ed1e638a995180a1c76e9232086229"}
Sep 11 16:02:42.777638 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.777620    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ng2ww_07d57c04-0697-419b-9d3e-ca11761da85c/ovn-acl-logging/0.log"
Sep 11 16:02:42.777991 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.777966    2577 generic.go:356] "Generic (PLEG): container finished" podID="07d57c04-0697-419b-9d3e-ca11761da85c" containerID="03a135dd0029d5e5f0bed99d70c1aaaf936b9c9c9da666bf9abcbb92d986663e" exitCode=1
Sep 11 16:02:42.778099 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.778067    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww" event={"ID":"07d57c04-0697-419b-9d3e-ca11761da85c","Type":"ContainerStarted","Data":"8e04c60d647a802f331acd3dd670672c29146a67a80c41078aa6f6235495bf86"}
Sep 11 16:02:42.778163 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.778146    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww" event={"ID":"07d57c04-0697-419b-9d3e-ca11761da85c","Type":"ContainerStarted","Data":"ecf50c838e9edcd693fc3b7e8a93b12345f25c5790dcd08aea9991fa8a98d9ca"}
Sep 11 16:02:42.778218 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.778195    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww" event={"ID":"07d57c04-0697-419b-9d3e-ca11761da85c","Type":"ContainerStarted","Data":"45e90b6dde9e480ea9425412bc77359c12f3c4cd8028e8a3774e5df3c1395ad7"}
Sep 11 16:02:42.778271 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.778242    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww" event={"ID":"07d57c04-0697-419b-9d3e-ca11761da85c","Type":"ContainerDied","Data":"03a135dd0029d5e5f0bed99d70c1aaaf936b9c9c9da666bf9abcbb92d986663e"}
Sep 11 16:02:42.778325 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.778293    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww" event={"ID":"07d57c04-0697-419b-9d3e-ca11761da85c","Type":"ContainerStarted","Data":"751ca19f74c5de878d43e9ea8642aaf7704ae7baf0423534829b8f9217d8a9fe"}
Sep 11 16:02:42.779603 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.779584    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mnrzt" event={"ID":"a790ffd9-de74-4a62-9e37-d0ab20455b53","Type":"ContainerStarted","Data":"a4430b15993276abe42c3ebf9162c6c176f62db5bd3192871640e52f2842b7aa"}
Sep 11 16:02:42.781058 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.781016    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d2xpr" event={"ID":"4c9f7e6e-73fc-4d05-b2a3-bffad79fa6cf","Type":"ContainerStarted","Data":"1324274fb8588afaf43c0899eb29b07669aaa6d63a7d394fd4c243ebb86890a1"}
Sep 11 16:02:42.782433 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.782412    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr" event={"ID":"12576b4c-228c-4ed2-ad49-3f8ab3a24462","Type":"ContainerStarted","Data":"1b55d90533f1452d3259c82a3905685fd870553c59dd1aef1bc8f7321c28e1fe"}
Sep 11 16:02:42.783876 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.783856    2577 generic.go:356] "Generic (PLEG): container finished" podID="86dab592-922f-41e5-bcfc-5edc18d28569" containerID="3318a67191fbe068e72643981c4da0f592ae9b44e76e3a2ec26a6b95f5336eef" exitCode=0
Sep 11 16:02:42.783966 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.783935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dn4s5" event={"ID":"86dab592-922f-41e5-bcfc-5edc18d28569","Type":"ContainerDied","Data":"3318a67191fbe068e72643981c4da0f592ae9b44e76e3a2ec26a6b95f5336eef"}
Sep 11 16:02:42.788629 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.788604    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ssjlf" event={"ID":"5f2f3c80-6520-4b83-9b00-04f95a2167b3","Type":"ContainerStarted","Data":"3c534e86fbdd12a5a635ca9c02b8b9d149662743081542b2c4871d51d900c8df"}
Sep 11 16:02:42.793833 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.793784    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.ec2.internal" podStartSLOduration=20.793756265 podStartE2EDuration="20.793756265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:02:25.755300958 +0000 UTC m=+5.600944518" watchObservedRunningTime="2026-09-11 16:02:42.793756265 +0000 UTC m=+22.639399797"
Sep 11 16:02:42.793928 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.793887    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-d68vg" podStartSLOduration=3.7428771960000002 podStartE2EDuration="21.793882299s" totalImagesPullingTime="18.051005103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:02:42.793731836 +0000 UTC m=+22.639375391" watchObservedRunningTime="2026-09-11 16:02:42.793882299 +0000 UTC m=+22.639525853"
Sep 11 16:02:42.811331 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.811286    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ssjlf" podStartSLOduration=3.809264212 podStartE2EDuration="21.811270422s" totalImagesPullingTime="18.00200621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:02:42.811192823 +0000 UTC m=+22.656836379" watchObservedRunningTime="2026-09-11 16:02:42.811270422 +0000 UTC m=+22.656913955"
Sep 11 16:02:42.831959 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.831916    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mnrzt" podStartSLOduration=3.936346509 podStartE2EDuration="21.831899254s" totalImagesPullingTime="17.895552745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:02:42.831248705 +0000 UTC m=+22.676892259" watchObservedRunningTime="2026-09-11 16:02:42.831899254 +0000 UTC m=+22.677542804"
Sep 11 16:02:42.851052 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.850985    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hlqbr" podStartSLOduration=3.9497219120000002 podStartE2EDuration="21.850969719s" totalImagesPullingTime="17.901247807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:02:42.850720944 +0000 UTC m=+22.696364503" watchObservedRunningTime="2026-09-11 16:02:42.850969719 +0000 UTC m=+22.696613271"
Sep 11 16:02:42.921079 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.921004    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d2xpr" podStartSLOduration=8.118950593 podStartE2EDuration="21.920985522s" totalImagesPullingTime="13.802034929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:02:42.873835452 +0000 UTC m=+22.719479008" watchObservedRunningTime="2026-09-11 16:02:42.920985522 +0000 UTC m=+22.766629076"
Sep 11 16:02:42.961022 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.960987    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d2xpr"
Sep 11 16:02:42.961587 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:42.961547    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d2xpr"
Sep 11 16:02:43.792314 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:43.792233    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pdmrf" event={"ID":"d0d91384-95ee-4acf-a00f-3becd929b46c","Type":"ContainerStarted","Data":"8615bb927063271f9711957c94819b649c9672a106e1cb5cba21a46e83d73ad5"}
Sep 11 16:02:43.795209 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:43.795182    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ng2ww_07d57c04-0697-419b-9d3e-ca11761da85c/ovn-acl-logging/0.log"
Sep 11 16:02:43.795670 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:43.795627    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww" event={"ID":"07d57c04-0697-419b-9d3e-ca11761da85c","Type":"ContainerStarted","Data":"28cd210df90aa5af9ff45183dffaf04fba62bd2c6f39d77fa78c41d67f0166a6"}
Sep 11 16:02:43.809735 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:43.809686    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pdmrf" podStartSLOduration=4.842023925 podStartE2EDuration="22.809665277s" totalImagesPullingTime="17.967641352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:02:43.809101987 +0000 UTC m=+23.654745542" watchObservedRunningTime="2026-09-11 16:02:43.809665277 +0000 UTC m=+23.655308833"
Sep 11 16:02:44.030955 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:44.030897    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 16:02:44.287979 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:44.287639    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d2xpr"
Sep 11 16:02:44.288357 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:44.288307    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d2xpr"
Sep 11 16:02:44.575140 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:44.574960    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T16:02:44.030959242Z","UUID":"efc051ce-1e4b-4fd5-9680-b6ca9c18a8c2","Handler":null,"Name":"","Endpoint":""}
Sep 11 16:02:44.576991 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:44.576763    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 16:02:44.576991 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:44.576798    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 16:02:44.627308 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:44.626852    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:44.627308 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:44.626872    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:44.627308 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:44.626982    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vw4vl" podUID="3c6a5b81-20d0-4f62-b730-356ee812b681"
Sep 11 16:02:44.627308 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:44.627055    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:44.627308 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:44.627147    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s8699" podUID="14f2dae2-6830-49c3-8f40-786b7f33ec2d"
Sep 11 16:02:44.627308 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:44.627221    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8j7q7" podUID="82491956-7b84-4719-b383-9dd3e987931b"
Sep 11 16:02:44.801203 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:44.801163    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr" event={"ID":"12576b4c-228c-4ed2-ad49-3f8ab3a24462","Type":"ContainerStarted","Data":"f7ae47219159253e43caaece6c04807332e81144c2a1ad047222ddc62da04d71"}
Sep 11 16:02:45.807298 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:45.807084    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ng2ww_07d57c04-0697-419b-9d3e-ca11761da85c/ovn-acl-logging/0.log"
Sep 11 16:02:45.807755 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:45.807698    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww" event={"ID":"07d57c04-0697-419b-9d3e-ca11761da85c","Type":"ContainerStarted","Data":"3a5576764de19ce6bb21b56a56f34b2b765f6c7003543a1d85040e3e865829fc"}
Sep 11 16:02:45.809669 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:45.809642    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr" event={"ID":"12576b4c-228c-4ed2-ad49-3f8ab3a24462","Type":"ContainerStarted","Data":"08e5aedb1a046b15371b2934bb801d92dfc7d4431b4074777a83513006b38405"}
Sep 11 16:02:45.828748 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:45.828688    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hdgnr" podStartSLOduration=3.9360896690000002 podStartE2EDuration="24.828667687s" totalImagesPullingTime="20.892578018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:02:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:02:45.828577129 +0000 UTC m=+25.674220685" watchObservedRunningTime="2026-09-11 16:02:45.828667687 +0000 UTC m=+25.674311244"
Sep 11 16:02:46.626437 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:46.626401    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:46.626614 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:46.626401    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:46.626614 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:46.626552    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vw4vl" podUID="3c6a5b81-20d0-4f62-b730-356ee812b681"
Sep 11 16:02:46.626614 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:46.626603    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:46.626776 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:46.626690    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8j7q7" podUID="82491956-7b84-4719-b383-9dd3e987931b"
Sep 11 16:02:46.626776 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:46.626758    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s8699" podUID="14f2dae2-6830-49c3-8f40-786b7f33ec2d"
Sep 11 16:02:47.818345 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:47.818150    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ng2ww_07d57c04-0697-419b-9d3e-ca11761da85c/ovn-acl-logging/0.log"
Sep 11 16:02:47.818723 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:47.818677    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww" event={"ID":"07d57c04-0697-419b-9d3e-ca11761da85c","Type":"ContainerStarted","Data":"5fb85e94f42352857ddcbfadc29355df2cd1c54f0ec772706a70cd42da0ad02f"}
Sep 11 16:02:47.818938 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:47.818921    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:47.819139 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:47.819113    2577 scope.go:117] "RemoveContainer" containerID="03a135dd0029d5e5f0bed99d70c1aaaf936b9c9c9da666bf9abcbb92d986663e"
Sep 11 16:02:47.820810 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:47.820784    2577 generic.go:356] "Generic (PLEG): container finished" podID="86dab592-922f-41e5-bcfc-5edc18d28569" containerID="c40c7ec9dc6cb8a047d914946ed15a90f35c2273d2e3264771353f4ce6cfa1b4" exitCode=0
Sep 11 16:02:47.820904 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:47.820832    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dn4s5" event={"ID":"86dab592-922f-41e5-bcfc-5edc18d28569","Type":"ContainerDied","Data":"c40c7ec9dc6cb8a047d914946ed15a90f35c2273d2e3264771353f4ce6cfa1b4"}
Sep 11 16:02:47.835885 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:47.835861    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:48.628099 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:48.628062    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:48.628253 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:48.628228    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vw4vl" podUID="3c6a5b81-20d0-4f62-b730-356ee812b681"
Sep 11 16:02:48.628654 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:48.628630    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:48.628748 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:48.628729    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8j7q7" podUID="82491956-7b84-4719-b383-9dd3e987931b"
Sep 11 16:02:48.628885 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:48.628868    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:48.628978 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:48.628959    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s8699" podUID="14f2dae2-6830-49c3-8f40-786b7f33ec2d"
Sep 11 16:02:48.826213 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:48.826179    2577 generic.go:356] "Generic (PLEG): container finished" podID="86dab592-922f-41e5-bcfc-5edc18d28569" containerID="0ea837a2698baa56a8f7b60169baccc604ec026d9aa60b45e1f3e6a31aac7c11" exitCode=0
Sep 11 16:02:48.826879 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:48.826258    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dn4s5" event={"ID":"86dab592-922f-41e5-bcfc-5edc18d28569","Type":"ContainerDied","Data":"0ea837a2698baa56a8f7b60169baccc604ec026d9aa60b45e1f3e6a31aac7c11"}
Sep 11 16:02:48.832445 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:48.832423    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ng2ww_07d57c04-0697-419b-9d3e-ca11761da85c/ovn-acl-logging/0.log"
Sep 11 16:02:48.832824 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:48.832802    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww" event={"ID":"07d57c04-0697-419b-9d3e-ca11761da85c","Type":"ContainerStarted","Data":"c764e63cf2fb12d1b3298704edf76a6eaed53c5bf745f096811012275ee934a6"}
Sep 11 16:02:48.833002 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:48.832987    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:48.833085 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:48.833029    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:48.849412 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:48.849365    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:02:48.879572 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:48.879524    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww" podStartSLOduration=9.844100212 podStartE2EDuration="27.879510017s" totalImagesPullingTime="18.035409805s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:02:48.877801869 +0000 UTC m=+28.723445437" watchObservedRunningTime="2026-09-11 16:02:48.879510017 +0000 UTC m=+28.725153569"
Sep 11 16:02:49.331540 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:49.331505    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vw4vl"]
Sep 11 16:02:49.331726 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:49.331687    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:49.331839 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:49.331811    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vw4vl" podUID="3c6a5b81-20d0-4f62-b730-356ee812b681"
Sep 11 16:02:49.335315 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:49.334792    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s8699"]
Sep 11 16:02:49.335315 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:49.334909    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:49.335315 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:49.334948    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8j7q7"]
Sep 11 16:02:49.335315 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:49.335008    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s8699" podUID="14f2dae2-6830-49c3-8f40-786b7f33ec2d"
Sep 11 16:02:49.335315 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:49.335055    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:49.335315 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:49.335139    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8j7q7" podUID="82491956-7b84-4719-b383-9dd3e987931b"
Sep 11 16:02:49.837141 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:49.837053    2577 generic.go:356] "Generic (PLEG): container finished" podID="86dab592-922f-41e5-bcfc-5edc18d28569" containerID="e36c431de32ceb350f94fd8171b65e7ad3ed08bd85ba29cea63babe338f3cefd" exitCode=0
Sep 11 16:02:49.837696 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:49.837141    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dn4s5" event={"ID":"86dab592-922f-41e5-bcfc-5edc18d28569","Type":"ContainerDied","Data":"e36c431de32ceb350f94fd8171b65e7ad3ed08bd85ba29cea63babe338f3cefd"}
Sep 11 16:02:50.628050 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:50.627751    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:50.628232 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:50.627829    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:50.628232 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:50.628178    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vw4vl" podUID="3c6a5b81-20d0-4f62-b730-356ee812b681"
Sep 11 16:02:50.628343 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:50.627849    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:50.628343 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:50.628259    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8j7q7" podUID="82491956-7b84-4719-b383-9dd3e987931b"
Sep 11 16:02:50.628434 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:50.628364    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s8699" podUID="14f2dae2-6830-49c3-8f40-786b7f33ec2d"
Sep 11 16:02:52.626638 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:52.626602    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:52.627098 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:52.626603    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:52.627098 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:52.626732    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:52.627098 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:52.626733    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8j7q7" podUID="82491956-7b84-4719-b383-9dd3e987931b"
Sep 11 16:02:52.627098 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:52.626862    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vw4vl" podUID="3c6a5b81-20d0-4f62-b730-356ee812b681"
Sep 11 16:02:52.627098 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:52.626971    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s8699" podUID="14f2dae2-6830-49c3-8f40-786b7f33ec2d"
Sep 11 16:02:54.267495 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:54.267157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mvdl\" (UniqueName: \"kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl\") pod \"network-check-target-8j7q7\" (UID: \"82491956-7b84-4719-b383-9dd3e987931b\") " pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:54.267495 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:54.267364    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:02:54.267495 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:54.267384    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:02:54.267495 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:54.267394    2577 projected.go:196] Error preparing data for projected volume kube-api-access-8mvdl for pod openshift-network-diagnostics/network-check-target-8j7q7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:02:54.267495 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:54.267450    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl podName:82491956-7b84-4719-b383-9dd3e987931b nodeName:}" failed. No retries permitted until 2026-09-11 16:03:26.267429664 +0000 UTC m=+66.113073198 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mvdl" (UniqueName: "kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl") pod "network-check-target-8j7q7" (UID: "82491956-7b84-4719-b383-9dd3e987931b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:02:54.367951 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:54.367918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs\") pod \"network-metrics-daemon-vw4vl\" (UID: \"3c6a5b81-20d0-4f62-b730-356ee812b681\") " pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:54.368147 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:54.367967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret\") pod \"global-pull-secret-syncer-s8699\" (UID: \"14f2dae2-6830-49c3-8f40-786b7f33ec2d\") " pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:54.368147 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:54.368101    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:02:54.368147 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:54.368101    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:02:54.368303 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:54.368169    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs podName:3c6a5b81-20d0-4f62-b730-356ee812b681 nodeName:}" failed. No retries permitted until 2026-09-11 16:03:26.368152051 +0000 UTC m=+66.213795604 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs") pod "network-metrics-daemon-vw4vl" (UID: "3c6a5b81-20d0-4f62-b730-356ee812b681") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:02:54.368303 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:54.368184    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret podName:14f2dae2-6830-49c3-8f40-786b7f33ec2d nodeName:}" failed. No retries permitted until 2026-09-11 16:03:26.368177243 +0000 UTC m=+66.213820775 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret") pod "global-pull-secret-syncer-s8699" (UID: "14f2dae2-6830-49c3-8f40-786b7f33ec2d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:02:54.626485 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:54.626456    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:54.626485 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:54.626486    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:54.626743 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:54.626456    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:54.626743 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:54.626563    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8j7q7" podUID="82491956-7b84-4719-b383-9dd3e987931b"
Sep 11 16:02:54.626743 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:54.626642    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vw4vl" podUID="3c6a5b81-20d0-4f62-b730-356ee812b681"
Sep 11 16:02:54.626743 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:54.626710    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s8699" podUID="14f2dae2-6830-49c3-8f40-786b7f33ec2d"
Sep 11 16:02:54.855427 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:54.855392    2577 generic.go:356] "Generic (PLEG): container finished" podID="86dab592-922f-41e5-bcfc-5edc18d28569" containerID="1ee3526c5a85a491120d6caf4b17ca4e293a13a089a2bd367f1d943badd04f94" exitCode=0
Sep 11 16:02:54.855628 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:54.855450    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dn4s5" event={"ID":"86dab592-922f-41e5-bcfc-5edc18d28569","Type":"ContainerDied","Data":"1ee3526c5a85a491120d6caf4b17ca4e293a13a089a2bd367f1d943badd04f94"}
Sep 11 16:02:55.861090 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:55.861059    2577 generic.go:356] "Generic (PLEG): container finished" podID="86dab592-922f-41e5-bcfc-5edc18d28569" containerID="7b441d6613292bed8504f41859c9634bfa037b64da3086930eea76c92d95b77b" exitCode=0
Sep 11 16:02:55.861480 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:55.861126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dn4s5" event={"ID":"86dab592-922f-41e5-bcfc-5edc18d28569","Type":"ContainerDied","Data":"7b441d6613292bed8504f41859c9634bfa037b64da3086930eea76c92d95b77b"}
Sep 11 16:02:55.942448 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:55.942424    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-140.ec2.internal" event="NodeReady"
Sep 11 16:02:55.942573 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:55.942545    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 16:02:55.995601 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:55.995424    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zqfcd"]
Sep 11 16:02:56.010681 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.010651    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-chv7q"]
Sep 11 16:02:56.010804 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.010725    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:02:56.013400 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.013374    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 16:02:56.013516 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.013418    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lgnfp\""
Sep 11 16:02:56.013516 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.013431    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 16:02:56.038655 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.038626    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zqfcd"]
Sep 11 16:02:56.038655 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.038659    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-chv7q"]
Sep 11 16:02:56.038856 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.038740    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-chv7q"
Sep 11 16:02:56.041526 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.041506    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 16:02:56.041657 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.041558    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 16:02:56.041657 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.041570    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 16:02:56.041657 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.041652    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gkgdt\""
Sep 11 16:02:56.180582 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.180496    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ba7b000-8761-42c8-9ef7-700cd3f1959f-cert\") pod \"ingress-canary-chv7q\" (UID: \"2ba7b000-8761-42c8-9ef7-700cd3f1959f\") " pod="openshift-ingress-canary/ingress-canary-chv7q"
Sep 11 16:02:56.180729 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.180600    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dafc1233-f30d-40de-9736-3cef5d49666f-tmp-dir\") pod \"dns-default-zqfcd\" (UID: \"dafc1233-f30d-40de-9736-3cef5d49666f\") " pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:02:56.180729 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.180624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4tdd\" (UniqueName: \"kubernetes.io/projected/dafc1233-f30d-40de-9736-3cef5d49666f-kube-api-access-r4tdd\") pod \"dns-default-zqfcd\" (UID: \"dafc1233-f30d-40de-9736-3cef5d49666f\") " pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:02:56.180729 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.180656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dafc1233-f30d-40de-9736-3cef5d49666f-config-volume\") pod \"dns-default-zqfcd\" (UID: \"dafc1233-f30d-40de-9736-3cef5d49666f\") " pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:02:56.180729 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.180679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn97b\" (UniqueName: \"kubernetes.io/projected/2ba7b000-8761-42c8-9ef7-700cd3f1959f-kube-api-access-xn97b\") pod \"ingress-canary-chv7q\" (UID: \"2ba7b000-8761-42c8-9ef7-700cd3f1959f\") " pod="openshift-ingress-canary/ingress-canary-chv7q"
Sep 11 16:02:56.180729 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.180696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls\") pod \"dns-default-zqfcd\" (UID: \"dafc1233-f30d-40de-9736-3cef5d49666f\") " pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:02:56.281861 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.281820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dafc1233-f30d-40de-9736-3cef5d49666f-config-volume\") pod \"dns-default-zqfcd\" (UID: \"dafc1233-f30d-40de-9736-3cef5d49666f\") " pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:02:56.281861 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.281861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn97b\" (UniqueName: \"kubernetes.io/projected/2ba7b000-8761-42c8-9ef7-700cd3f1959f-kube-api-access-xn97b\") pod \"ingress-canary-chv7q\" (UID: \"2ba7b000-8761-42c8-9ef7-700cd3f1959f\") " pod="openshift-ingress-canary/ingress-canary-chv7q"
Sep 11 16:02:56.282155 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.281881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls\") pod \"dns-default-zqfcd\" (UID: \"dafc1233-f30d-40de-9736-3cef5d49666f\") " pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:02:56.282155 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.281899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ba7b000-8761-42c8-9ef7-700cd3f1959f-cert\") pod \"ingress-canary-chv7q\" (UID: \"2ba7b000-8761-42c8-9ef7-700cd3f1959f\") " pod="openshift-ingress-canary/ingress-canary-chv7q"
Sep 11 16:02:56.282155 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:56.282000    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:02:56.282155 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:56.282096    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls podName:dafc1233-f30d-40de-9736-3cef5d49666f nodeName:}" failed. No retries permitted until 2026-09-11 16:02:56.782073987 +0000 UTC m=+36.627717534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls") pod "dns-default-zqfcd" (UID: "dafc1233-f30d-40de-9736-3cef5d49666f") : secret "dns-default-metrics-tls" not found
Sep 11 16:02:56.282155 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.282143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dafc1233-f30d-40de-9736-3cef5d49666f-tmp-dir\") pod \"dns-default-zqfcd\" (UID: \"dafc1233-f30d-40de-9736-3cef5d49666f\") " pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:02:56.282453 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.282174    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4tdd\" (UniqueName: \"kubernetes.io/projected/dafc1233-f30d-40de-9736-3cef5d49666f-kube-api-access-r4tdd\") pod \"dns-default-zqfcd\" (UID: \"dafc1233-f30d-40de-9736-3cef5d49666f\") " pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:02:56.282453 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.282442    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dafc1233-f30d-40de-9736-3cef5d49666f-tmp-dir\") pod \"dns-default-zqfcd\" (UID: \"dafc1233-f30d-40de-9736-3cef5d49666f\") " pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:02:56.282571 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.282551    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dafc1233-f30d-40de-9736-3cef5d49666f-config-volume\") pod \"dns-default-zqfcd\" (UID: \"dafc1233-f30d-40de-9736-3cef5d49666f\") " pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:02:56.286640 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.286617    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ba7b000-8761-42c8-9ef7-700cd3f1959f-cert\") pod \"ingress-canary-chv7q\" (UID: \"2ba7b000-8761-42c8-9ef7-700cd3f1959f\") " pod="openshift-ingress-canary/ingress-canary-chv7q"
Sep 11 16:02:56.291587 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.291554    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4tdd\" (UniqueName: \"kubernetes.io/projected/dafc1233-f30d-40de-9736-3cef5d49666f-kube-api-access-r4tdd\") pod \"dns-default-zqfcd\" (UID: \"dafc1233-f30d-40de-9736-3cef5d49666f\") " pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:02:56.291733 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.291712    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn97b\" (UniqueName: \"kubernetes.io/projected/2ba7b000-8761-42c8-9ef7-700cd3f1959f-kube-api-access-xn97b\") pod \"ingress-canary-chv7q\" (UID: \"2ba7b000-8761-42c8-9ef7-700cd3f1959f\") " pod="openshift-ingress-canary/ingress-canary-chv7q"
Sep 11 16:02:56.347804 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.347767    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-chv7q"
Sep 11 16:02:56.626882 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.626847    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:02:56.627060 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.626850    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:02:56.627060 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.626851    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:02:56.630898 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.630873    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 16:02:56.632077 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.632057    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rkwl5\""
Sep 11 16:02:56.632413 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.632392    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 16:02:56.632492 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.632420    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 16:02:56.632492 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.632437    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 16:02:56.632492 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.632421    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tt876\""
Sep 11 16:02:56.665535 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.665509    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-chv7q"]
Sep 11 16:02:56.668806 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:02:56.668784    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba7b000_8761_42c8_9ef7_700cd3f1959f.slice/crio-404c6a23ed94a6d917200822bf756ffadedc635851966a178cc7b26bbd081f1d WatchSource:0}: Error finding container 404c6a23ed94a6d917200822bf756ffadedc635851966a178cc7b26bbd081f1d: Status 404 returned error can't find the container with id 404c6a23ed94a6d917200822bf756ffadedc635851966a178cc7b26bbd081f1d
Sep 11 16:02:56.786438 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.786348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls\") pod \"dns-default-zqfcd\" (UID: \"dafc1233-f30d-40de-9736-3cef5d49666f\") " pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:02:56.786603 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:56.786466    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:02:56.786603 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:56.786522    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls podName:dafc1233-f30d-40de-9736-3cef5d49666f nodeName:}" failed. No retries permitted until 2026-09-11 16:02:57.786506114 +0000 UTC m=+37.632149667 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls") pod "dns-default-zqfcd" (UID: "dafc1233-f30d-40de-9736-3cef5d49666f") : secret "dns-default-metrics-tls" not found
Sep 11 16:02:56.866839 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.866800    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dn4s5" event={"ID":"86dab592-922f-41e5-bcfc-5edc18d28569","Type":"ContainerStarted","Data":"44f2bd327975f7469b7b8a16e4b2c47b75148d61c19947a58ba1452029c6c186"}
Sep 11 16:02:56.867984 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.867966    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-chv7q" event={"ID":"2ba7b000-8761-42c8-9ef7-700cd3f1959f","Type":"ContainerStarted","Data":"404c6a23ed94a6d917200822bf756ffadedc635851966a178cc7b26bbd081f1d"}
Sep 11 16:02:56.891275 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:56.891218    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dn4s5" podStartSLOduration=7.365584255 podStartE2EDuration="35.891200176s" totalImagesPullingTime="28.525615921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:02:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:02:56.88974183 +0000 UTC m=+36.735385384" watchObservedRunningTime="2026-09-11 16:02:56.891200176 +0000 UTC m=+36.736843725"
Sep 11 16:02:57.792810 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:57.792776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls\") pod \"dns-default-zqfcd\" (UID: \"dafc1233-f30d-40de-9736-3cef5d49666f\") " pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:02:57.792964 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:57.792933    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:02:57.793014 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:57.793003    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls podName:dafc1233-f30d-40de-9736-3cef5d49666f nodeName:}" failed. No retries permitted until 2026-09-11 16:02:59.79298524 +0000 UTC m=+39.638628803 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls") pod "dns-default-zqfcd" (UID: "dafc1233-f30d-40de-9736-3cef5d49666f") : secret "dns-default-metrics-tls" not found
Sep 11 16:02:59.808018 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:02:59.807981    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls\") pod \"dns-default-zqfcd\" (UID: \"dafc1233-f30d-40de-9736-3cef5d49666f\") " pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:02:59.808506 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:59.808164    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:02:59.808506 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:02:59.808239    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls podName:dafc1233-f30d-40de-9736-3cef5d49666f nodeName:}" failed. No retries permitted until 2026-09-11 16:03:03.80822152 +0000 UTC m=+43.653865053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls") pod "dns-default-zqfcd" (UID: "dafc1233-f30d-40de-9736-3cef5d49666f") : secret "dns-default-metrics-tls" not found
Sep 11 16:03:00.879962 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:00.879929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-chv7q" event={"ID":"2ba7b000-8761-42c8-9ef7-700cd3f1959f","Type":"ContainerStarted","Data":"144d9e363dbfd04330bb278c6effe76c03257b5d451d86d1c836e7cd0d2f4a58"}
Sep 11 16:03:00.897586 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:00.897530    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-chv7q" podStartSLOduration=2.201562529 podStartE2EDuration="5.897513232s" totalImagesPullingTime="3.695950703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:00.896755399 +0000 UTC m=+40.742398955" watchObservedRunningTime="2026-09-11 16:03:00.897513232 +0000 UTC m=+40.743156787"
Sep 11 16:03:01.228374 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.228298    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-v5mzm"]
Sep 11 16:03:01.243304 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.243278    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-v5mzm"]
Sep 11 16:03:01.243453 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.243370    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-v5mzm"
Sep 11 16:03:01.246356 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.246332    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 11 16:03:01.246356 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.246354    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tj68s\""
Sep 11 16:03:01.246561 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.246340    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 11 16:03:01.246659 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.246645    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 11 16:03:01.247317 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.247295    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:03:01.252717 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.252697    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 11 16:03:01.332518 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.332483    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-6vx4t"]
Sep 11 16:03:01.353364 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.353328    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-6vx4t"]
Sep 11 16:03:01.353562 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.353424    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-6vx4t"
Sep 11 16:03:01.356921 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.356472    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 11 16:03:01.356921 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.356477    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 11 16:03:01.356921 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.356768    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wc4k9\""
Sep 11 16:03:01.357174 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.357156    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 11 16:03:01.357302 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.357287    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:03:01.419997 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.419958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/286b445a-e066-44bb-8e69-cd3b0d0ccb30-trusted-ca\") pod \"console-operator-696884fb9c-v5mzm\" (UID: \"286b445a-e066-44bb-8e69-cd3b0d0ccb30\") " pod="openshift-console-operator/console-operator-696884fb9c-v5mzm"
Sep 11 16:03:01.420178 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.420019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg74c\" (UniqueName: \"kubernetes.io/projected/286b445a-e066-44bb-8e69-cd3b0d0ccb30-kube-api-access-wg74c\") pod \"console-operator-696884fb9c-v5mzm\" (UID: \"286b445a-e066-44bb-8e69-cd3b0d0ccb30\") " pod="openshift-console-operator/console-operator-696884fb9c-v5mzm"
Sep 11 16:03:01.420178 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.420093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/286b445a-e066-44bb-8e69-cd3b0d0ccb30-config\") pod \"console-operator-696884fb9c-v5mzm\" (UID: \"286b445a-e066-44bb-8e69-cd3b0d0ccb30\") " pod="openshift-console-operator/console-operator-696884fb9c-v5mzm"
Sep 11 16:03:01.420178 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.420135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/286b445a-e066-44bb-8e69-cd3b0d0ccb30-serving-cert\") pod \"console-operator-696884fb9c-v5mzm\" (UID: \"286b445a-e066-44bb-8e69-cd3b0d0ccb30\") " pod="openshift-console-operator/console-operator-696884fb9c-v5mzm"
Sep 11 16:03:01.444072 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.444026    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fmmjp"]
Sep 11 16:03:01.457414 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.457383    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fmmjp"
Sep 11 16:03:01.460458 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.460441    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-bsnlh\""
Sep 11 16:03:01.460721 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.460707    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 11 16:03:01.460940 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.460928    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:03:01.486343 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.486272    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fmmjp"]
Sep 11 16:03:01.520463 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.520435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/286b445a-e066-44bb-8e69-cd3b0d0ccb30-trusted-ca\") pod \"console-operator-696884fb9c-v5mzm\" (UID: \"286b445a-e066-44bb-8e69-cd3b0d0ccb30\") " pod="openshift-console-operator/console-operator-696884fb9c-v5mzm"
Sep 11 16:03:01.520625 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.520473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8sbk\" (UniqueName: \"kubernetes.io/projected/a8658a10-11a7-4fda-99bb-dcd369d5918c-kube-api-access-b8sbk\") pod \"service-ca-operator-cf7c776b7-6vx4t\" (UID: \"a8658a10-11a7-4fda-99bb-dcd369d5918c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-6vx4t"
Sep 11 16:03:01.520625 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.520508    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg74c\" (UniqueName: \"kubernetes.io/projected/286b445a-e066-44bb-8e69-cd3b0d0ccb30-kube-api-access-wg74c\") pod \"console-operator-696884fb9c-v5mzm\" (UID: \"286b445a-e066-44bb-8e69-cd3b0d0ccb30\") " pod="openshift-console-operator/console-operator-696884fb9c-v5mzm"
Sep 11 16:03:01.520625 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.520571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/286b445a-e066-44bb-8e69-cd3b0d0ccb30-config\") pod \"console-operator-696884fb9c-v5mzm\" (UID: \"286b445a-e066-44bb-8e69-cd3b0d0ccb30\") " pod="openshift-console-operator/console-operator-696884fb9c-v5mzm"
Sep 11 16:03:01.520625 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.520621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/286b445a-e066-44bb-8e69-cd3b0d0ccb30-serving-cert\") pod \"console-operator-696884fb9c-v5mzm\" (UID: \"286b445a-e066-44bb-8e69-cd3b0d0ccb30\") " pod="openshift-console-operator/console-operator-696884fb9c-v5mzm"
Sep 11 16:03:01.520782 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.520639    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8658a10-11a7-4fda-99bb-dcd369d5918c-config\") pod \"service-ca-operator-cf7c776b7-6vx4t\" (UID: \"a8658a10-11a7-4fda-99bb-dcd369d5918c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-6vx4t"
Sep 11 16:03:01.520782 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.520676    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8658a10-11a7-4fda-99bb-dcd369d5918c-serving-cert\") pod \"service-ca-operator-cf7c776b7-6vx4t\" (UID: \"a8658a10-11a7-4fda-99bb-dcd369d5918c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-6vx4t"
Sep 11 16:03:01.521388 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.521365    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/286b445a-e066-44bb-8e69-cd3b0d0ccb30-config\") pod \"console-operator-696884fb9c-v5mzm\" (UID: \"286b445a-e066-44bb-8e69-cd3b0d0ccb30\") " pod="openshift-console-operator/console-operator-696884fb9c-v5mzm"
Sep 11 16:03:01.521388 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.521381    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/286b445a-e066-44bb-8e69-cd3b0d0ccb30-trusted-ca\") pod \"console-operator-696884fb9c-v5mzm\" (UID: \"286b445a-e066-44bb-8e69-cd3b0d0ccb30\") " pod="openshift-console-operator/console-operator-696884fb9c-v5mzm"
Sep 11 16:03:01.523407 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.523390    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/286b445a-e066-44bb-8e69-cd3b0d0ccb30-serving-cert\") pod \"console-operator-696884fb9c-v5mzm\" (UID: \"286b445a-e066-44bb-8e69-cd3b0d0ccb30\") " pod="openshift-console-operator/console-operator-696884fb9c-v5mzm"
Sep 11 16:03:01.529152 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.529129    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg74c\" (UniqueName: \"kubernetes.io/projected/286b445a-e066-44bb-8e69-cd3b0d0ccb30-kube-api-access-wg74c\") pod \"console-operator-696884fb9c-v5mzm\" (UID: \"286b445a-e066-44bb-8e69-cd3b0d0ccb30\") " pod="openshift-console-operator/console-operator-696884fb9c-v5mzm"
Sep 11 16:03:01.553131 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.553097    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-v5mzm"
Sep 11 16:03:01.621625 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.621586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8658a10-11a7-4fda-99bb-dcd369d5918c-serving-cert\") pod \"service-ca-operator-cf7c776b7-6vx4t\" (UID: \"a8658a10-11a7-4fda-99bb-dcd369d5918c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-6vx4t"
Sep 11 16:03:01.621753 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.621651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b8js\" (UniqueName: \"kubernetes.io/projected/16e8c31c-1064-4b5b-a1c9-1d2e509f6853-kube-api-access-9b8js\") pod \"volume-data-source-validator-86695b7cc7-fmmjp\" (UID: \"16e8c31c-1064-4b5b-a1c9-1d2e509f6853\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fmmjp"
Sep 11 16:03:01.621753 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.621689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8sbk\" (UniqueName: \"kubernetes.io/projected/a8658a10-11a7-4fda-99bb-dcd369d5918c-kube-api-access-b8sbk\") pod \"service-ca-operator-cf7c776b7-6vx4t\" (UID: \"a8658a10-11a7-4fda-99bb-dcd369d5918c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-6vx4t"
Sep 11 16:03:01.621902 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.621878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8658a10-11a7-4fda-99bb-dcd369d5918c-config\") pod \"service-ca-operator-cf7c776b7-6vx4t\" (UID: \"a8658a10-11a7-4fda-99bb-dcd369d5918c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-6vx4t"
Sep 11 16:03:01.622665 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.622637    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8658a10-11a7-4fda-99bb-dcd369d5918c-config\") pod \"service-ca-operator-cf7c776b7-6vx4t\" (UID: \"a8658a10-11a7-4fda-99bb-dcd369d5918c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-6vx4t"
Sep 11 16:03:01.626514 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.625591    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8658a10-11a7-4fda-99bb-dcd369d5918c-serving-cert\") pod \"service-ca-operator-cf7c776b7-6vx4t\" (UID: \"a8658a10-11a7-4fda-99bb-dcd369d5918c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-6vx4t"
Sep 11 16:03:01.629322 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.629299    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8sbk\" (UniqueName: \"kubernetes.io/projected/a8658a10-11a7-4fda-99bb-dcd369d5918c-kube-api-access-b8sbk\") pod \"service-ca-operator-cf7c776b7-6vx4t\" (UID: \"a8658a10-11a7-4fda-99bb-dcd369d5918c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-6vx4t"
Sep 11 16:03:01.663327 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.663286    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-6vx4t"
Sep 11 16:03:01.723103 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.723067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b8js\" (UniqueName: \"kubernetes.io/projected/16e8c31c-1064-4b5b-a1c9-1d2e509f6853-kube-api-access-9b8js\") pod \"volume-data-source-validator-86695b7cc7-fmmjp\" (UID: \"16e8c31c-1064-4b5b-a1c9-1d2e509f6853\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fmmjp"
Sep 11 16:03:01.731696 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.731668    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b8js\" (UniqueName: \"kubernetes.io/projected/16e8c31c-1064-4b5b-a1c9-1d2e509f6853-kube-api-access-9b8js\") pod \"volume-data-source-validator-86695b7cc7-fmmjp\" (UID: \"16e8c31c-1064-4b5b-a1c9-1d2e509f6853\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fmmjp"
Sep 11 16:03:01.767553 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.767521    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fmmjp"
Sep 11 16:03:01.844579 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.844540    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-v5mzm"]
Sep 11 16:03:01.883701 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.883668    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-v5mzm" event={"ID":"286b445a-e066-44bb-8e69-cd3b0d0ccb30","Type":"ContainerStarted","Data":"6eaa7f7ee0097e4e2b6dd0b9a5658af2e222c89966afe03ce92a56289a3f064e"}
Sep 11 16:03:01.959519 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:01.959489    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-6vx4t"]
Sep 11 16:03:01.962519 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:03:01.962485    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8658a10_11a7_4fda_99bb_dcd369d5918c.slice/crio-aa43c79bcbbb77263f81d62868b018895347456badca1ac2b02f3c866b9de459 WatchSource:0}: Error finding container aa43c79bcbbb77263f81d62868b018895347456badca1ac2b02f3c866b9de459: Status 404 returned error can't find the container with id aa43c79bcbbb77263f81d62868b018895347456badca1ac2b02f3c866b9de459
Sep 11 16:03:02.062216 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:02.062132    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fmmjp"]
Sep 11 16:03:02.065194 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:03:02.065167    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e8c31c_1064_4b5b_a1c9_1d2e509f6853.slice/crio-e3f2e0e3bd6153dbc402f39a6566e903292e1b0156d690aea1b3aeca034aef77 WatchSource:0}: Error finding container e3f2e0e3bd6153dbc402f39a6566e903292e1b0156d690aea1b3aeca034aef77: Status 404 returned error can't find the container with id e3f2e0e3bd6153dbc402f39a6566e903292e1b0156d690aea1b3aeca034aef77
Sep 11 16:03:02.887530 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:02.887496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-6vx4t" event={"ID":"a8658a10-11a7-4fda-99bb-dcd369d5918c","Type":"ContainerStarted","Data":"aa43c79bcbbb77263f81d62868b018895347456badca1ac2b02f3c866b9de459"}
Sep 11 16:03:02.888755 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:02.888725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fmmjp" event={"ID":"16e8c31c-1064-4b5b-a1c9-1d2e509f6853","Type":"ContainerStarted","Data":"e3f2e0e3bd6153dbc402f39a6566e903292e1b0156d690aea1b3aeca034aef77"}
Sep 11 16:03:03.837470 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:03.837433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls\") pod \"dns-default-zqfcd\" (UID: \"dafc1233-f30d-40de-9736-3cef5d49666f\") " pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:03:03.837634 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:03.837611    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:03:03.837705 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:03.837685    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls podName:dafc1233-f30d-40de-9736-3cef5d49666f nodeName:}" failed. No retries permitted until 2026-09-11 16:03:11.837668175 +0000 UTC m=+51.683311709 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls") pod "dns-default-zqfcd" (UID: "dafc1233-f30d-40de-9736-3cef5d49666f") : secret "dns-default-metrics-tls" not found
Sep 11 16:03:03.894079 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:03.894021    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fmmjp" event={"ID":"16e8c31c-1064-4b5b-a1c9-1d2e509f6853","Type":"ContainerStarted","Data":"11140bd80e018784d16d81baf1c08f0b63b208a4b8229cdb1cc4b6576cdd9aab"}
Sep 11 16:03:03.914478 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:03.914413    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-fmmjp" podStartSLOduration=1.6436389409999999 podStartE2EDuration="2.914389005s" totalImagesPullingTime="1.270750064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:03:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:03.912659452 +0000 UTC m=+43.758303011" watchObservedRunningTime="2026-09-11 16:03:03.914389005 +0000 UTC m=+43.760032561"
Sep 11 16:03:05.629592 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.629547    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c9g8w"]
Sep 11 16:03:05.631909 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.631888    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:05.634678 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.634646    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 16:03:05.634805 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.634745    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 16:03:05.634805 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.634749    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 16:03:05.635671 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.635655    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 16:03:05.635782 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.635711    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-krpkn\""
Sep 11 16:03:05.653298 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.653272    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c9g8w"]
Sep 11 16:03:05.752930 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.752885    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/07825a72-a9e3-47f9-a430-a64b648c5dbb-data-volume\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:05.752930 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.752936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:05.753202 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.753022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07825a72-a9e3-47f9-a430-a64b648c5dbb-crio-socket\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:05.753202 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.753128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:05.753202 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.753149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqwxn\" (UniqueName: \"kubernetes.io/projected/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-api-access-zqwxn\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:05.854286 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.854256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/07825a72-a9e3-47f9-a430-a64b648c5dbb-data-volume\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:05.854286 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.854290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:05.854492 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.854325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07825a72-a9e3-47f9-a430-a64b648c5dbb-crio-socket\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:05.854492 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.854386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:05.854492 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:05.854407    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:03:05.854604 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:05.854501    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-rbac-proxy-cm podName:07825a72-a9e3-47f9-a430-a64b648c5dbb nodeName:}" failed. No retries permitted until 2026-09-11 16:03:06.354479467 +0000 UTC m=+46.200123015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c9g8w" (UID: "07825a72-a9e3-47f9-a430-a64b648c5dbb") : configmap "kube-rbac-proxy" not found
Sep 11 16:03:05.854604 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:05.854508    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:03:05.854604 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:05.854557    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls podName:07825a72-a9e3-47f9-a430-a64b648c5dbb nodeName:}" failed. No retries permitted until 2026-09-11 16:03:06.354545825 +0000 UTC m=+46.200189383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c9g8w" (UID: "07825a72-a9e3-47f9-a430-a64b648c5dbb") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:03:05.854604 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.854411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqwxn\" (UniqueName: \"kubernetes.io/projected/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-api-access-zqwxn\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:05.854604 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.854579    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07825a72-a9e3-47f9-a430-a64b648c5dbb-crio-socket\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:05.854762 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.854643    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/07825a72-a9e3-47f9-a430-a64b648c5dbb-data-volume\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:05.864306 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.864275    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqwxn\" (UniqueName: \"kubernetes.io/projected/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-api-access-zqwxn\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:05.900000 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.899910    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-6vx4t" event={"ID":"a8658a10-11a7-4fda-99bb-dcd369d5918c","Type":"ContainerStarted","Data":"40c376e0857a86587d9206e5fe8aed3604eb35f329dda3ac2e9c614551127168"}
Sep 11 16:03:05.901628 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.901606    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/0.log"
Sep 11 16:03:05.901768 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.901642    2577 generic.go:356] "Generic (PLEG): container finished" podID="286b445a-e066-44bb-8e69-cd3b0d0ccb30" containerID="6fbe92a383330ec780f620a09ffec9dea83f71d8bc69f1d77f12440fe648e607" exitCode=255
Sep 11 16:03:05.901768 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.901683    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-v5mzm" event={"ID":"286b445a-e066-44bb-8e69-cd3b0d0ccb30","Type":"ContainerDied","Data":"6fbe92a383330ec780f620a09ffec9dea83f71d8bc69f1d77f12440fe648e607"}
Sep 11 16:03:05.901994 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.901976    2577 scope.go:117] "RemoveContainer" containerID="6fbe92a383330ec780f620a09ffec9dea83f71d8bc69f1d77f12440fe648e607"
Sep 11 16:03:05.916867 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:05.916814    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-6vx4t" podStartSLOduration=1.9982858430000001 podStartE2EDuration="4.916795177s" totalImagesPullingTime="2.918509334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:03:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:05.91651471 +0000 UTC m=+45.762158265" watchObservedRunningTime="2026-09-11 16:03:05.916795177 +0000 UTC m=+45.762438733"
Sep 11 16:03:06.358386 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:06.358353    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:06.358534 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:06.358433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:06.358534 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:06.358471    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:03:06.358534 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:06.358525    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:03:06.358640 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:06.358537    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-rbac-proxy-cm podName:07825a72-a9e3-47f9-a430-a64b648c5dbb nodeName:}" failed. No retries permitted until 2026-09-11 16:03:07.358518487 +0000 UTC m=+47.204162021 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c9g8w" (UID: "07825a72-a9e3-47f9-a430-a64b648c5dbb") : configmap "kube-rbac-proxy" not found
Sep 11 16:03:06.358640 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:06.358573    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls podName:07825a72-a9e3-47f9-a430-a64b648c5dbb nodeName:}" failed. No retries permitted until 2026-09-11 16:03:07.358561927 +0000 UTC m=+47.204205460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c9g8w" (UID: "07825a72-a9e3-47f9-a430-a64b648c5dbb") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:03:06.907382 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:06.907353    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/1.log"
Sep 11 16:03:06.907777 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:06.907755    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/0.log"
Sep 11 16:03:06.907887 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:06.907798    2577 generic.go:356] "Generic (PLEG): container finished" podID="286b445a-e066-44bb-8e69-cd3b0d0ccb30" containerID="c174517cbdc818c4976c17f7ebd0f9c365bcdbd45666e59fdb3cb1b36ec3a3f7" exitCode=255
Sep 11 16:03:06.907925 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:06.907890    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-v5mzm" event={"ID":"286b445a-e066-44bb-8e69-cd3b0d0ccb30","Type":"ContainerDied","Data":"c174517cbdc818c4976c17f7ebd0f9c365bcdbd45666e59fdb3cb1b36ec3a3f7"}
Sep 11 16:03:06.907964 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:06.907950    2577 scope.go:117] "RemoveContainer" containerID="6fbe92a383330ec780f620a09ffec9dea83f71d8bc69f1d77f12440fe648e607"
Sep 11 16:03:06.908239 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:06.908222    2577 scope.go:117] "RemoveContainer" containerID="c174517cbdc818c4976c17f7ebd0f9c365bcdbd45666e59fdb3cb1b36ec3a3f7"
Sep 11 16:03:06.908432 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:06.908410    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-v5mzm_openshift-console-operator(286b445a-e066-44bb-8e69-cd3b0d0ccb30)\"" pod="openshift-console-operator/console-operator-696884fb9c-v5mzm" podUID="286b445a-e066-44bb-8e69-cd3b0d0ccb30"
Sep 11 16:03:07.361686 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:07.361658    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mnrzt_a790ffd9-de74-4a62-9e37-d0ab20455b53/dns-node-resolver/0.log"
Sep 11 16:03:07.367342 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:07.367314    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:07.367553 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:07.367536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:07.367849 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:07.367831    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:03:07.368000 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:07.367990    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-rbac-proxy-cm podName:07825a72-a9e3-47f9-a430-a64b648c5dbb nodeName:}" failed. No retries permitted until 2026-09-11 16:03:09.367971188 +0000 UTC m=+49.213614750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c9g8w" (UID: "07825a72-a9e3-47f9-a430-a64b648c5dbb") : configmap "kube-rbac-proxy" not found
Sep 11 16:03:07.368615 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:07.368596    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:03:07.368771 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:07.368761    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls podName:07825a72-a9e3-47f9-a430-a64b648c5dbb nodeName:}" failed. No retries permitted until 2026-09-11 16:03:09.368734674 +0000 UTC m=+49.214378224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c9g8w" (UID: "07825a72-a9e3-47f9-a430-a64b648c5dbb") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:03:07.581209 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:07.581132    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-n8nf2"]
Sep 11 16:03:07.585860 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:07.585842    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-n8nf2"
Sep 11 16:03:07.589126 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:07.589101    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2khhf\""
Sep 11 16:03:07.589260 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:07.589218    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 11 16:03:07.589329 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:07.589278    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 11 16:03:07.602827 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:07.602798    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-n8nf2"]
Sep 11 16:03:07.670454 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:07.670414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrd2v\" (UniqueName: \"kubernetes.io/projected/7ee154d1-43a0-47a1-8c27-a66b55cacec1-kube-api-access-lrd2v\") pod \"migrator-8b4d697c5-n8nf2\" (UID: \"7ee154d1-43a0-47a1-8c27-a66b55cacec1\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-n8nf2"
Sep 11 16:03:07.771121 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:07.771081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrd2v\" (UniqueName: \"kubernetes.io/projected/7ee154d1-43a0-47a1-8c27-a66b55cacec1-kube-api-access-lrd2v\") pod \"migrator-8b4d697c5-n8nf2\" (UID: \"7ee154d1-43a0-47a1-8c27-a66b55cacec1\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-n8nf2"
Sep 11 16:03:07.784346 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:07.784315    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrd2v\" (UniqueName: \"kubernetes.io/projected/7ee154d1-43a0-47a1-8c27-a66b55cacec1-kube-api-access-lrd2v\") pod \"migrator-8b4d697c5-n8nf2\" (UID: \"7ee154d1-43a0-47a1-8c27-a66b55cacec1\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-n8nf2"
Sep 11 16:03:07.894228 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:07.894186    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-n8nf2"
Sep 11 16:03:07.912384 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:07.912357    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/1.log"
Sep 11 16:03:07.912982 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:07.912653    2577 scope.go:117] "RemoveContainer" containerID="c174517cbdc818c4976c17f7ebd0f9c365bcdbd45666e59fdb3cb1b36ec3a3f7"
Sep 11 16:03:07.912982 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:07.912796    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-v5mzm_openshift-console-operator(286b445a-e066-44bb-8e69-cd3b0d0ccb30)\"" pod="openshift-console-operator/console-operator-696884fb9c-v5mzm" podUID="286b445a-e066-44bb-8e69-cd3b0d0ccb30"
Sep 11 16:03:08.188697 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:08.188622    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-n8nf2"]
Sep 11 16:03:08.192215 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:03:08.192171    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee154d1_43a0_47a1_8c27_a66b55cacec1.slice/crio-e8ea5d12198a2c44c92125bc65fc076bde309d5314c7dcd9aa0497dcbf56480e WatchSource:0}: Error finding container e8ea5d12198a2c44c92125bc65fc076bde309d5314c7dcd9aa0497dcbf56480e: Status 404 returned error can't find the container with id e8ea5d12198a2c44c92125bc65fc076bde309d5314c7dcd9aa0497dcbf56480e
Sep 11 16:03:08.764023 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:08.763999    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hlqbr_58ea2e2b-bb7b-4537-aebf-68daed411f86/node-ca/0.log"
Sep 11 16:03:08.916458 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:08.916423    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-n8nf2" event={"ID":"7ee154d1-43a0-47a1-8c27-a66b55cacec1","Type":"ContainerStarted","Data":"e8ea5d12198a2c44c92125bc65fc076bde309d5314c7dcd9aa0497dcbf56480e"}
Sep 11 16:03:09.165057 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:09.165011    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-chv7q_2ba7b000-8761-42c8-9ef7-700cd3f1959f/serve-healthcheck-canary/0.log"
Sep 11 16:03:09.383130 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:09.383090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:09.383275 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:09.383156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:09.383275 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:09.383229    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:03:09.383275 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:09.383246    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:03:09.383389 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:09.383283    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-rbac-proxy-cm podName:07825a72-a9e3-47f9-a430-a64b648c5dbb nodeName:}" failed. No retries permitted until 2026-09-11 16:03:13.38326956 +0000 UTC m=+53.228913093 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c9g8w" (UID: "07825a72-a9e3-47f9-a430-a64b648c5dbb") : configmap "kube-rbac-proxy" not found
Sep 11 16:03:09.383389 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:09.383301    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls podName:07825a72-a9e3-47f9-a430-a64b648c5dbb nodeName:}" failed. No retries permitted until 2026-09-11 16:03:13.383288273 +0000 UTC m=+53.228931810 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c9g8w" (UID: "07825a72-a9e3-47f9-a430-a64b648c5dbb") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:03:09.921325 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:09.921294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-n8nf2" event={"ID":"7ee154d1-43a0-47a1-8c27-a66b55cacec1","Type":"ContainerStarted","Data":"735254e463e2953efacf30e9f45d57c1e8e30b3eb7e43ae9dbaf0be869ecb7e2"}
Sep 11 16:03:09.921693 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:09.921341    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-n8nf2" event={"ID":"7ee154d1-43a0-47a1-8c27-a66b55cacec1","Type":"ContainerStarted","Data":"61679a4ed35d8df7855102ecde2cd539fcc9c2d77fa6452c891871f6e019e7c3"}
Sep 11 16:03:09.950243 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:09.950140    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-n8nf2" podStartSLOduration=1.46985538 podStartE2EDuration="2.950120949s" totalImagesPullingTime="1.480265569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:09.949308778 +0000 UTC m=+49.794952333" watchObservedRunningTime="2026-09-11 16:03:09.950120949 +0000 UTC m=+49.795764512"
Sep 11 16:03:11.553333 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:11.553288    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-v5mzm"
Sep 11 16:03:11.553775 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:11.553353    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-v5mzm"
Sep 11 16:03:11.553775 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:11.553727    2577 scope.go:117] "RemoveContainer" containerID="c174517cbdc818c4976c17f7ebd0f9c365bcdbd45666e59fdb3cb1b36ec3a3f7"
Sep 11 16:03:11.553922 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:11.553902    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-v5mzm_openshift-console-operator(286b445a-e066-44bb-8e69-cd3b0d0ccb30)\"" pod="openshift-console-operator/console-operator-696884fb9c-v5mzm" podUID="286b445a-e066-44bb-8e69-cd3b0d0ccb30"
Sep 11 16:03:11.903937 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:11.903902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls\") pod \"dns-default-zqfcd\" (UID: \"dafc1233-f30d-40de-9736-3cef5d49666f\") " pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:03:11.904243 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:11.904078    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:03:11.904243 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:11.904142    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls podName:dafc1233-f30d-40de-9736-3cef5d49666f nodeName:}" failed. No retries permitted until 2026-09-11 16:03:27.904123968 +0000 UTC m=+67.749767500 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls") pod "dns-default-zqfcd" (UID: "dafc1233-f30d-40de-9736-3cef5d49666f") : secret "dns-default-metrics-tls" not found
Sep 11 16:03:13.415429 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:13.415388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:13.415931 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:13.415451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:13.415931 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:13.415530    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:03:13.415931 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:13.415568    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:03:13.415931 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:13.415601    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-rbac-proxy-cm podName:07825a72-a9e3-47f9-a430-a64b648c5dbb nodeName:}" failed. No retries permitted until 2026-09-11 16:03:21.415585572 +0000 UTC m=+61.261229125 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c9g8w" (UID: "07825a72-a9e3-47f9-a430-a64b648c5dbb") : configmap "kube-rbac-proxy" not found
Sep 11 16:03:13.415931 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:13.415636    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls podName:07825a72-a9e3-47f9-a430-a64b648c5dbb nodeName:}" failed. No retries permitted until 2026-09-11 16:03:21.415617156 +0000 UTC m=+61.261260688 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c9g8w" (UID: "07825a72-a9e3-47f9-a430-a64b648c5dbb") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:03:20.851196 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:20.851169    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ng2ww"
Sep 11 16:03:21.477870 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:21.477821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:21.478065 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:21.477915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:21.478065 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:21.477974    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:03:21.478143 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:21.478071    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls podName:07825a72-a9e3-47f9-a430-a64b648c5dbb nodeName:}" failed. No retries permitted until 2026-09-11 16:03:37.478028168 +0000 UTC m=+77.323671702 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c9g8w" (UID: "07825a72-a9e3-47f9-a430-a64b648c5dbb") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:03:21.478521 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:21.478502    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07825a72-a9e3-47f9-a430-a64b648c5dbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:24.626469 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:24.626433    2577 scope.go:117] "RemoveContainer" containerID="c174517cbdc818c4976c17f7ebd0f9c365bcdbd45666e59fdb3cb1b36ec3a3f7"
Sep 11 16:03:24.967517 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:24.967442    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/1.log"
Sep 11 16:03:24.967663 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:24.967519    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-v5mzm" event={"ID":"286b445a-e066-44bb-8e69-cd3b0d0ccb30","Type":"ContainerStarted","Data":"7e0924d96959e000e55dde8a6465e0c1b5944114934073ed2707c3ae39217a78"}
Sep 11 16:03:24.967799 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:24.967785    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-v5mzm"
Sep 11 16:03:25.004000 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:25.003931    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-v5mzm" podStartSLOduration=20.972445174 podStartE2EDuration="24.003891808s" totalImagesPullingTime="3.031446634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:03:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:25.003601308 +0000 UTC m=+64.849244864" watchObservedRunningTime="2026-09-11 16:03:25.003891808 +0000 UTC m=+64.849535364"
Sep 11 16:03:25.249765 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:25.249690    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-v5mzm"
Sep 11 16:03:26.313945 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.313913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mvdl\" (UniqueName: \"kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl\") pod \"network-check-target-8j7q7\" (UID: \"82491956-7b84-4719-b383-9dd3e987931b\") " pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:03:26.316861 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.316837    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 16:03:26.326879 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.326855    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 16:03:26.337435 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.337408    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mvdl\" (UniqueName: \"kubernetes.io/projected/82491956-7b84-4719-b383-9dd3e987931b-kube-api-access-8mvdl\") pod \"network-check-target-8j7q7\" (UID: \"82491956-7b84-4719-b383-9dd3e987931b\") " pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:03:26.348230 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.348199    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rkwl5\""
Sep 11 16:03:26.355517 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.355493    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:03:26.414590 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.414553    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret\") pod \"global-pull-secret-syncer-s8699\" (UID: \"14f2dae2-6830-49c3-8f40-786b7f33ec2d\") " pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:03:26.414743 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.414641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs\") pod \"network-metrics-daemon-vw4vl\" (UID: \"3c6a5b81-20d0-4f62-b730-356ee812b681\") " pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:03:26.418017 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.417856    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 16:03:26.418017 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.417980    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 16:03:26.427431 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.427381    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6a5b81-20d0-4f62-b730-356ee812b681-metrics-certs\") pod \"network-metrics-daemon-vw4vl\" (UID: \"3c6a5b81-20d0-4f62-b730-356ee812b681\") " pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:03:26.427431 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.427414    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14f2dae2-6830-49c3-8f40-786b7f33ec2d-original-pull-secret\") pod \"global-pull-secret-syncer-s8699\" (UID: \"14f2dae2-6830-49c3-8f40-786b7f33ec2d\") " pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:03:26.629468 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:03:26.629436    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82491956_7b84_4719_b383_9dd3e987931b.slice/crio-d90b5939878bc6e5165104757b248e73287830e0445a5b0041a5f3fa2859d655 WatchSource:0}: Error finding container d90b5939878bc6e5165104757b248e73287830e0445a5b0041a5f3fa2859d655: Status 404 returned error can't find the container with id d90b5939878bc6e5165104757b248e73287830e0445a5b0041a5f3fa2859d655
Sep 11 16:03:26.629751 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.629731    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8j7q7"]
Sep 11 16:03:26.639725 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.639703    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s8699"
Sep 11 16:03:26.654332 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.654312    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tt876\""
Sep 11 16:03:26.661879 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.661854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vw4vl"
Sep 11 16:03:26.919247 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.919128    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s8699"]
Sep 11 16:03:26.921605 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:03:26.921581    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f2dae2_6830_49c3_8f40_786b7f33ec2d.slice/crio-922cd65a6aed008f9ec90ac377e85048b95c0d602e1e5f785719a611dcee3915 WatchSource:0}: Error finding container 922cd65a6aed008f9ec90ac377e85048b95c0d602e1e5f785719a611dcee3915: Status 404 returned error can't find the container with id 922cd65a6aed008f9ec90ac377e85048b95c0d602e1e5f785719a611dcee3915
Sep 11 16:03:26.941501 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.941469    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vw4vl"]
Sep 11 16:03:26.944306 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:03:26.944271    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6a5b81_20d0_4f62_b730_356ee812b681.slice/crio-c44b9eafe85edd3b9a90c308f7d5221eb2738a11c90b7f6703559046c0770d7b WatchSource:0}: Error finding container c44b9eafe85edd3b9a90c308f7d5221eb2738a11c90b7f6703559046c0770d7b: Status 404 returned error can't find the container with id c44b9eafe85edd3b9a90c308f7d5221eb2738a11c90b7f6703559046c0770d7b
Sep 11 16:03:26.974734 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.974700    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vw4vl" event={"ID":"3c6a5b81-20d0-4f62-b730-356ee812b681","Type":"ContainerStarted","Data":"c44b9eafe85edd3b9a90c308f7d5221eb2738a11c90b7f6703559046c0770d7b"}
Sep 11 16:03:26.975832 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.975806    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s8699" event={"ID":"14f2dae2-6830-49c3-8f40-786b7f33ec2d","Type":"ContainerStarted","Data":"922cd65a6aed008f9ec90ac377e85048b95c0d602e1e5f785719a611dcee3915"}
Sep 11 16:03:26.976865 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:26.976847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8j7q7" event={"ID":"82491956-7b84-4719-b383-9dd3e987931b","Type":"ContainerStarted","Data":"d90b5939878bc6e5165104757b248e73287830e0445a5b0041a5f3fa2859d655"}
Sep 11 16:03:27.926356 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:27.926328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls\") pod \"dns-default-zqfcd\" (UID: \"dafc1233-f30d-40de-9736-3cef5d49666f\") " pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:03:27.929961 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:27.929935    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dafc1233-f30d-40de-9736-3cef5d49666f-metrics-tls\") pod \"dns-default-zqfcd\" (UID: \"dafc1233-f30d-40de-9736-3cef5d49666f\") " pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:03:28.126525 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:28.126496    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lgnfp\""
Sep 11 16:03:28.131895 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:28.131866    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:03:28.438766 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:28.438731    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zqfcd"]
Sep 11 16:03:28.985199 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:28.985162    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zqfcd" event={"ID":"dafc1233-f30d-40de-9736-3cef5d49666f","Type":"ContainerStarted","Data":"7347ac0d1fd2a4dce5972a46b1bc6c35d3b4db8ca3ce327fa4b2d941d310e163"}
Sep 11 16:03:28.987842 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:28.987801    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vw4vl" event={"ID":"3c6a5b81-20d0-4f62-b730-356ee812b681","Type":"ContainerStarted","Data":"679391fd2645e8155746b5fc2ad7e765173a07fa94018197ef37d67d22a146fe"}
Sep 11 16:03:28.987981 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:28.987867    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vw4vl" event={"ID":"3c6a5b81-20d0-4f62-b730-356ee812b681","Type":"ContainerStarted","Data":"9cd6d937527064421a7a6212ecb3bc1554b6f6cd4fedd1b554bd61e43654df50"}
Sep 11 16:03:29.010869 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:29.010804    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vw4vl" podStartSLOduration=67.040536871 podStartE2EDuration="1m8.010780672s" totalImagesPullingTime="970.243801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:29.009950136 +0000 UTC m=+68.855593691" watchObservedRunningTime="2026-09-11 16:03:29.010780672 +0000 UTC m=+68.856424227"
Sep 11 16:03:32.824226 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.824194    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8bd59d6b-gvv5v"]
Sep 11 16:03:32.826252 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.826229    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8bd59d6b-gvv5v"
Sep 11 16:03:32.831200 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.831172    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 16:03:32.831414 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.831396    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 16:03:32.832012 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.831858    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 16:03:32.832012 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.831891    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-x7hqs\""
Sep 11 16:03:32.832012 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.831902    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 16:03:32.839491 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.839464    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"]
Sep 11 16:03:32.841754 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.841728    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8bd59d6b-gvv5v"]
Sep 11 16:03:32.841890 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.841851    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"
Sep 11 16:03:32.847256 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.847234    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 16:03:32.847805 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.847779    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 16:03:32.847931 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.847807    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 16:03:32.847931 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.847864    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 16:03:32.867208 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.867178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p78ns\" (UniqueName: \"kubernetes.io/projected/ba457642-bde8-4b0a-9583-e03a2ef75ed9-kube-api-access-p78ns\") pod \"managed-serviceaccount-addon-agent-8bd59d6b-gvv5v\" (UID: \"ba457642-bde8-4b0a-9583-e03a2ef75ed9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8bd59d6b-gvv5v"
Sep 11 16:03:32.867363 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.867233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ba457642-bde8-4b0a-9583-e03a2ef75ed9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8bd59d6b-gvv5v\" (UID: \"ba457642-bde8-4b0a-9583-e03a2ef75ed9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8bd59d6b-gvv5v"
Sep 11 16:03:32.879856 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.879826    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"]
Sep 11 16:03:32.968453 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.968418    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ba457642-bde8-4b0a-9583-e03a2ef75ed9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8bd59d6b-gvv5v\" (UID: \"ba457642-bde8-4b0a-9583-e03a2ef75ed9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8bd59d6b-gvv5v"
Sep 11 16:03:32.968453 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.968452    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/254e9993-04f9-4aed-8a75-77424f46b444-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77955d774-rtgkk\" (UID: \"254e9993-04f9-4aed-8a75-77424f46b444\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"
Sep 11 16:03:32.968694 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.968478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/254e9993-04f9-4aed-8a75-77424f46b444-ca\") pod \"cluster-proxy-proxy-agent-77955d774-rtgkk\" (UID: \"254e9993-04f9-4aed-8a75-77424f46b444\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"
Sep 11 16:03:32.968694 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.968537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/254e9993-04f9-4aed-8a75-77424f46b444-hub\") pod \"cluster-proxy-proxy-agent-77955d774-rtgkk\" (UID: \"254e9993-04f9-4aed-8a75-77424f46b444\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"
Sep 11 16:03:32.968694 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.968625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p78ns\" (UniqueName: \"kubernetes.io/projected/ba457642-bde8-4b0a-9583-e03a2ef75ed9-kube-api-access-p78ns\") pod \"managed-serviceaccount-addon-agent-8bd59d6b-gvv5v\" (UID: \"ba457642-bde8-4b0a-9583-e03a2ef75ed9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8bd59d6b-gvv5v"
Sep 11 16:03:32.968694 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.968666    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/254e9993-04f9-4aed-8a75-77424f46b444-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77955d774-rtgkk\" (UID: \"254e9993-04f9-4aed-8a75-77424f46b444\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"
Sep 11 16:03:32.968871 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.968699    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/254e9993-04f9-4aed-8a75-77424f46b444-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77955d774-rtgkk\" (UID: \"254e9993-04f9-4aed-8a75-77424f46b444\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"
Sep 11 16:03:32.968871 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.968728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rzjq\" (UniqueName: \"kubernetes.io/projected/254e9993-04f9-4aed-8a75-77424f46b444-kube-api-access-9rzjq\") pod \"cluster-proxy-proxy-agent-77955d774-rtgkk\" (UID: \"254e9993-04f9-4aed-8a75-77424f46b444\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"
Sep 11 16:03:32.971249 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.971223    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ba457642-bde8-4b0a-9583-e03a2ef75ed9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8bd59d6b-gvv5v\" (UID: \"ba457642-bde8-4b0a-9583-e03a2ef75ed9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8bd59d6b-gvv5v"
Sep 11 16:03:32.991671 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:32.991640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p78ns\" (UniqueName: \"kubernetes.io/projected/ba457642-bde8-4b0a-9583-e03a2ef75ed9-kube-api-access-p78ns\") pod \"managed-serviceaccount-addon-agent-8bd59d6b-gvv5v\" (UID: \"ba457642-bde8-4b0a-9583-e03a2ef75ed9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8bd59d6b-gvv5v"
Sep 11 16:03:33.003181 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.003151    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s8699" event={"ID":"14f2dae2-6830-49c3-8f40-786b7f33ec2d","Type":"ContainerStarted","Data":"6e425249f5b7732a8e08768ea9a6798bbdc175c2731f9ca0d3f33f447d731bdf"}
Sep 11 16:03:33.004681 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.004655    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8j7q7" event={"ID":"82491956-7b84-4719-b383-9dd3e987931b","Type":"ContainerStarted","Data":"03907dd1bc9cccb311e6d02bf2d88dcfb83d73702c1e3bda144f2cc8301e9283"}
Sep 11 16:03:33.004806 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.004764    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:03:33.006484 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.006465    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zqfcd" event={"ID":"dafc1233-f30d-40de-9736-3cef5d49666f","Type":"ContainerStarted","Data":"78f052e01560a81196759555de88e20dde0857fde5df225cdf44a0803150a182"}
Sep 11 16:03:33.006583 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.006498    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zqfcd" event={"ID":"dafc1233-f30d-40de-9736-3cef5d49666f","Type":"ContainerStarted","Data":"a2a8a01a58b2722367fde5471e84ed04a9cd3a65c9c11d95204b083736b5a915"}
Sep 11 16:03:33.006630 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.006611    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:03:33.034401 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.034346    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-s8699" podStartSLOduration=66.909712231 podStartE2EDuration="1m12.034329743s" totalImagesPullingTime="5.124617512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:33.03370308 +0000 UTC m=+72.879346635" watchObservedRunningTime="2026-09-11 16:03:33.034329743 +0000 UTC m=+72.879973314"
Sep 11 16:03:33.069234 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.069196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/254e9993-04f9-4aed-8a75-77424f46b444-ca\") pod \"cluster-proxy-proxy-agent-77955d774-rtgkk\" (UID: \"254e9993-04f9-4aed-8a75-77424f46b444\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"
Sep 11 16:03:33.069234 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.069233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/254e9993-04f9-4aed-8a75-77424f46b444-hub\") pod \"cluster-proxy-proxy-agent-77955d774-rtgkk\" (UID: \"254e9993-04f9-4aed-8a75-77424f46b444\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"
Sep 11 16:03:33.069419 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.069379    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/254e9993-04f9-4aed-8a75-77424f46b444-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77955d774-rtgkk\" (UID: \"254e9993-04f9-4aed-8a75-77424f46b444\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"
Sep 11 16:03:33.069472 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.069420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/254e9993-04f9-4aed-8a75-77424f46b444-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77955d774-rtgkk\" (UID: \"254e9993-04f9-4aed-8a75-77424f46b444\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"
Sep 11 16:03:33.069472 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.069454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rzjq\" (UniqueName: \"kubernetes.io/projected/254e9993-04f9-4aed-8a75-77424f46b444-kube-api-access-9rzjq\") pod \"cluster-proxy-proxy-agent-77955d774-rtgkk\" (UID: \"254e9993-04f9-4aed-8a75-77424f46b444\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"
Sep 11 16:03:33.069578 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.069543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/254e9993-04f9-4aed-8a75-77424f46b444-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77955d774-rtgkk\" (UID: \"254e9993-04f9-4aed-8a75-77424f46b444\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"
Sep 11 16:03:33.070446 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.070106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/254e9993-04f9-4aed-8a75-77424f46b444-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77955d774-rtgkk\" (UID: \"254e9993-04f9-4aed-8a75-77424f46b444\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"
Sep 11 16:03:33.072069 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.071064    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zqfcd" podStartSLOduration=34.367570122 podStartE2EDuration="38.071024838s" totalImagesPullingTime="3.703454716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:33.070083364 +0000 UTC m=+72.915726920" watchObservedRunningTime="2026-09-11 16:03:33.071024838 +0000 UTC m=+72.916668393"
Sep 11 16:03:33.072418 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.072396    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/254e9993-04f9-4aed-8a75-77424f46b444-ca\") pod \"cluster-proxy-proxy-agent-77955d774-rtgkk\" (UID: \"254e9993-04f9-4aed-8a75-77424f46b444\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"
Sep 11 16:03:33.072663 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.072636    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/254e9993-04f9-4aed-8a75-77424f46b444-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77955d774-rtgkk\" (UID: \"254e9993-04f9-4aed-8a75-77424f46b444\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"
Sep 11 16:03:33.073065 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.073019    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/254e9993-04f9-4aed-8a75-77424f46b444-hub\") pod \"cluster-proxy-proxy-agent-77955d774-rtgkk\" (UID: \"254e9993-04f9-4aed-8a75-77424f46b444\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"
Sep 11 16:03:33.074261 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.074216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/254e9993-04f9-4aed-8a75-77424f46b444-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77955d774-rtgkk\" (UID: \"254e9993-04f9-4aed-8a75-77424f46b444\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"
Sep 11 16:03:33.095398 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.095370    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rzjq\" (UniqueName: \"kubernetes.io/projected/254e9993-04f9-4aed-8a75-77424f46b444-kube-api-access-9rzjq\") pod \"cluster-proxy-proxy-agent-77955d774-rtgkk\" (UID: \"254e9993-04f9-4aed-8a75-77424f46b444\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"
Sep 11 16:03:33.108371 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.108313    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8j7q7" podStartSLOduration=66.702506284 podStartE2EDuration="1m12.108291522s" totalImagesPullingTime="5.405785238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:33.107786609 +0000 UTC m=+72.953430163" watchObservedRunningTime="2026-09-11 16:03:33.108291522 +0000 UTC m=+72.953935077"
Sep 11 16:03:33.148908 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.148874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8bd59d6b-gvv5v"
Sep 11 16:03:33.156358 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.156276    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"
Sep 11 16:03:33.426885 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.426807    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8bd59d6b-gvv5v"]
Sep 11 16:03:33.429793 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:03:33.429763    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba457642_bde8_4b0a_9583_e03a2ef75ed9.slice/crio-4ee186c1ceb331bf661f4de5137c148e1b50cea886627fe5838d3524c8908f94 WatchSource:0}: Error finding container 4ee186c1ceb331bf661f4de5137c148e1b50cea886627fe5838d3524c8908f94: Status 404 returned error can't find the container with id 4ee186c1ceb331bf661f4de5137c148e1b50cea886627fe5838d3524c8908f94
Sep 11 16:03:33.454361 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:33.454336    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk"]
Sep 11 16:03:33.456635 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:03:33.456604    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod254e9993_04f9_4aed_8a75_77424f46b444.slice/crio-6a4c983a0ac4063518440dc46aed7b8a065de39d216486fd01fe1b9ab60a1b0f WatchSource:0}: Error finding container 6a4c983a0ac4063518440dc46aed7b8a065de39d216486fd01fe1b9ab60a1b0f: Status 404 returned error can't find the container with id 6a4c983a0ac4063518440dc46aed7b8a065de39d216486fd01fe1b9ab60a1b0f
Sep 11 16:03:34.011123 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:34.011087    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk" event={"ID":"254e9993-04f9-4aed-8a75-77424f46b444","Type":"ContainerStarted","Data":"6a4c983a0ac4063518440dc46aed7b8a065de39d216486fd01fe1b9ab60a1b0f"}
Sep 11 16:03:34.012243 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:34.012218    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8bd59d6b-gvv5v" event={"ID":"ba457642-bde8-4b0a-9583-e03a2ef75ed9","Type":"ContainerStarted","Data":"4ee186c1ceb331bf661f4de5137c148e1b50cea886627fe5838d3524c8908f94"}
Sep 11 16:03:37.502570 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:37.502527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:37.504915 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:37.504894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07825a72-a9e3-47f9-a430-a64b648c5dbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9g8w\" (UID: \"07825a72-a9e3-47f9-a430-a64b648c5dbb\") " pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:37.743152 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:37.743119    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-krpkn\""
Sep 11 16:03:37.750499 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:37.750475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c9g8w"
Sep 11 16:03:38.032316 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:38.032202    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk" event={"ID":"254e9993-04f9-4aed-8a75-77424f46b444","Type":"ContainerStarted","Data":"a329fa3c69eac0e8c4e2ca576c5b13cddd2c9634940e32c354ecaf2bf24db2c3"}
Sep 11 16:03:38.032316 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:38.032288    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk" event={"ID":"254e9993-04f9-4aed-8a75-77424f46b444","Type":"ContainerStarted","Data":"704ed0038d7cbc2465d5951f52a386d9f312c25a74db7257c879ed356968350a"}
Sep 11 16:03:38.032316 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:38.032314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk" event={"ID":"254e9993-04f9-4aed-8a75-77424f46b444","Type":"ContainerStarted","Data":"5a3535abcab82ab4e4cca71d9f145b0d42f58558a0cadc4a3dd33631c208509b"}
Sep 11 16:03:38.033861 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:38.033830    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8bd59d6b-gvv5v" event={"ID":"ba457642-bde8-4b0a-9583-e03a2ef75ed9","Type":"ContainerStarted","Data":"9dafdc55f0fb3a659e7032dc513a3658f965518ff5f92d62d54438398624b74e"}
Sep 11 16:03:38.056402 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:38.056372    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c9g8w"]
Sep 11 16:03:38.058771 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:03:38.058738    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07825a72_a9e3_47f9_a430_a64b648c5dbb.slice/crio-213a3d20940f6089a4b2408dc4dd96a5ee6deff414bba905516fead664901eb6 WatchSource:0}: Error finding container 213a3d20940f6089a4b2408dc4dd96a5ee6deff414bba905516fead664901eb6: Status 404 returned error can't find the container with id 213a3d20940f6089a4b2408dc4dd96a5ee6deff414bba905516fead664901eb6
Sep 11 16:03:38.066817 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:38.066772    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77955d774-rtgkk" podStartSLOduration=2.220963836 podStartE2EDuration="6.066756633s" totalImagesPullingTime="3.845792797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:38.066206089 +0000 UTC m=+77.911849643" watchObservedRunningTime="2026-09-11 16:03:38.066756633 +0000 UTC m=+77.912400188"
Sep 11 16:03:38.091423 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:38.091370    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8bd59d6b-gvv5v" podStartSLOduration=2.229570117 podStartE2EDuration="6.091353799s" totalImagesPullingTime="3.861783682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:38.089252431 +0000 UTC m=+77.934895986" watchObservedRunningTime="2026-09-11 16:03:38.091353799 +0000 UTC m=+77.936997346"
Sep 11 16:03:39.040142 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:39.040047    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c9g8w" event={"ID":"07825a72-a9e3-47f9-a430-a64b648c5dbb","Type":"ContainerStarted","Data":"70bc0b465c1adfaad0ea3227bacefef8787ad9b38b5985399e199ce2c99e3367"}
Sep 11 16:03:39.040142 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:39.040106    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c9g8w" event={"ID":"07825a72-a9e3-47f9-a430-a64b648c5dbb","Type":"ContainerStarted","Data":"587b03d00face6035dee3dbc7abf02eb9399a411bc1ed30ef5c2cdc76d95b5a2"}
Sep 11 16:03:39.040142 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:39.040126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c9g8w" event={"ID":"07825a72-a9e3-47f9-a430-a64b648c5dbb","Type":"ContainerStarted","Data":"213a3d20940f6089a4b2408dc4dd96a5ee6deff414bba905516fead664901eb6"}
Sep 11 16:03:41.048118 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:41.048084    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c9g8w" event={"ID":"07825a72-a9e3-47f9-a430-a64b648c5dbb","Type":"ContainerStarted","Data":"d07a0242f75d0118b6c5ee3bfcf75fde72552013f5ffc24bad87931a929b6e16"}
Sep 11 16:03:41.070493 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:41.070432    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-c9g8w" podStartSLOduration=33.329102495 podStartE2EDuration="36.070415076s" totalImagesPullingTime="2.741312581s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:03:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:41.070383461 +0000 UTC m=+80.916027016" watchObservedRunningTime="2026-09-11 16:03:41.070415076 +0000 UTC m=+80.916058630"
Sep 11 16:03:43.015050 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:43.014921    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zqfcd"
Sep 11 16:03:47.662155 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.662117    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l8dfk"]
Sep 11 16:03:47.668088 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.668067    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.671403 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.671378    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 16:03:47.671549 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.671531    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 16:03:47.671601 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.671534    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2mr97\""
Sep 11 16:03:47.672223 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.672205    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 16:03:47.672608 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.672594    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 16:03:47.672864 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.672850    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 16:03:47.672978 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.672963    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 16:03:47.775052 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.774995    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57c804d5-daad-45c9-96c0-488161d776da-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.775052 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.775029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lhxd\" (UniqueName: \"kubernetes.io/projected/57c804d5-daad-45c9-96c0-488161d776da-kube-api-access-5lhxd\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.775270 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.775101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/57c804d5-daad-45c9-96c0-488161d776da-node-exporter-textfile\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.775270 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.775119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/57c804d5-daad-45c9-96c0-488161d776da-node-exporter-tls\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.775270 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.775177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/57c804d5-daad-45c9-96c0-488161d776da-node-exporter-wtmp\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.775270 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.775227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/57c804d5-daad-45c9-96c0-488161d776da-root\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.775270 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.775247    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/57c804d5-daad-45c9-96c0-488161d776da-node-exporter-accelerators-collector-config\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.775270 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.775272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57c804d5-daad-45c9-96c0-488161d776da-sys\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.775462 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.775294    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57c804d5-daad-45c9-96c0-488161d776da-metrics-client-ca\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.875648 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.875614    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/57c804d5-daad-45c9-96c0-488161d776da-root\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.875648 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.875647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/57c804d5-daad-45c9-96c0-488161d776da-node-exporter-accelerators-collector-config\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.875868 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.875671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57c804d5-daad-45c9-96c0-488161d776da-sys\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.875868 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.875693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57c804d5-daad-45c9-96c0-488161d776da-metrics-client-ca\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.875868 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.875739    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/57c804d5-daad-45c9-96c0-488161d776da-root\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.875868 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.875769    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57c804d5-daad-45c9-96c0-488161d776da-sys\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.875868 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.875802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57c804d5-daad-45c9-96c0-488161d776da-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.875868 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.875836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lhxd\" (UniqueName: \"kubernetes.io/projected/57c804d5-daad-45c9-96c0-488161d776da-kube-api-access-5lhxd\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.875868 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.875865    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/57c804d5-daad-45c9-96c0-488161d776da-node-exporter-textfile\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.876205 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.875887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/57c804d5-daad-45c9-96c0-488161d776da-node-exporter-tls\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.876205 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.875905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/57c804d5-daad-45c9-96c0-488161d776da-node-exporter-wtmp\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.876205 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:47.876072    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 16:03:47.876205 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.876097    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/57c804d5-daad-45c9-96c0-488161d776da-node-exporter-wtmp\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.876205 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:03:47.876132    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57c804d5-daad-45c9-96c0-488161d776da-node-exporter-tls podName:57c804d5-daad-45c9-96c0-488161d776da nodeName:}" failed. No retries permitted until 2026-09-11 16:03:48.376113551 +0000 UTC m=+88.221757087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/57c804d5-daad-45c9-96c0-488161d776da-node-exporter-tls") pod "node-exporter-l8dfk" (UID: "57c804d5-daad-45c9-96c0-488161d776da") : secret "node-exporter-tls" not found
Sep 11 16:03:47.876418 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.876213    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/57c804d5-daad-45c9-96c0-488161d776da-node-exporter-textfile\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.876418 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.876300    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/57c804d5-daad-45c9-96c0-488161d776da-node-exporter-accelerators-collector-config\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.876418 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.876315    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57c804d5-daad-45c9-96c0-488161d776da-metrics-client-ca\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.878361 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.878330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57c804d5-daad-45c9-96c0-488161d776da-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:47.890515 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:47.890484    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lhxd\" (UniqueName: \"kubernetes.io/projected/57c804d5-daad-45c9-96c0-488161d776da-kube-api-access-5lhxd\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:48.379918 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:48.379886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/57c804d5-daad-45c9-96c0-488161d776da-node-exporter-tls\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:48.382476 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:48.382449    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/57c804d5-daad-45c9-96c0-488161d776da-node-exporter-tls\") pod \"node-exporter-l8dfk\" (UID: \"57c804d5-daad-45c9-96c0-488161d776da\") " pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:48.577807 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:48.577769    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l8dfk"
Sep 11 16:03:48.590463 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:03:48.590426    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c804d5_daad_45c9_96c0_488161d776da.slice/crio-eb786e4dd7c1efe75765ec2c30cc677450ba8ca79d5773c7bb19df9c4918ab61 WatchSource:0}: Error finding container eb786e4dd7c1efe75765ec2c30cc677450ba8ca79d5773c7bb19df9c4918ab61: Status 404 returned error can't find the container with id eb786e4dd7c1efe75765ec2c30cc677450ba8ca79d5773c7bb19df9c4918ab61
Sep 11 16:03:49.074835 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:49.074798    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l8dfk" event={"ID":"57c804d5-daad-45c9-96c0-488161d776da","Type":"ContainerStarted","Data":"eb786e4dd7c1efe75765ec2c30cc677450ba8ca79d5773c7bb19df9c4918ab61"}
Sep 11 16:03:50.079981 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:50.079882    2577 generic.go:356] "Generic (PLEG): container finished" podID="57c804d5-daad-45c9-96c0-488161d776da" containerID="4eb146cfacbd6550d81161809fb1d484b63e10594106e206a1b0cef3eb42a8a3" exitCode=0
Sep 11 16:03:50.079981 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:50.079922    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l8dfk" event={"ID":"57c804d5-daad-45c9-96c0-488161d776da","Type":"ContainerDied","Data":"4eb146cfacbd6550d81161809fb1d484b63e10594106e206a1b0cef3eb42a8a3"}
Sep 11 16:03:51.085541 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:51.085501    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l8dfk" event={"ID":"57c804d5-daad-45c9-96c0-488161d776da","Type":"ContainerStarted","Data":"60a8fcd727d5e1996a922946bcaf8862f48c845e6815ac8982a63247dd8955df"}
Sep 11 16:03:51.085966 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:51.085563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l8dfk" event={"ID":"57c804d5-daad-45c9-96c0-488161d776da","Type":"ContainerStarted","Data":"91a00740292b841844065050064905b6bd30552a5f3174fb0c643348ded02bba"}
Sep 11 16:03:51.106391 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:03:51.106326    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l8dfk" podStartSLOduration=2.9338600919999998 podStartE2EDuration="4.10631025s" totalImagesPullingTime="1.10537054s" totalInitContainerRuntime="67.079618ms" isStatefulPod=true podCreationTimestamp="2026-09-11 16:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:51.106269199 +0000 UTC m=+90.951912754" watchObservedRunningTime="2026-09-11 16:03:51.10631025 +0000 UTC m=+90.951953804"
Sep 11 16:04:04.015054 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:04:04.015005    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8j7q7"
Sep 11 16:04:11.735959 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:04:11.735922    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-chv7q_2ba7b000-8761-42c8-9ef7-700cd3f1959f/serve-healthcheck-canary/0.log"
Sep 11 16:04:31.225705 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:04:31.225668    2577 generic.go:356] "Generic (PLEG): container finished" podID="a8658a10-11a7-4fda-99bb-dcd369d5918c" containerID="40c376e0857a86587d9206e5fe8aed3604eb35f329dda3ac2e9c614551127168" exitCode=0
Sep 11 16:04:31.226228 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:04:31.225740    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-6vx4t" event={"ID":"a8658a10-11a7-4fda-99bb-dcd369d5918c","Type":"ContainerDied","Data":"40c376e0857a86587d9206e5fe8aed3604eb35f329dda3ac2e9c614551127168"}
Sep 11 16:04:31.226228 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:04:31.226093    2577 scope.go:117] "RemoveContainer" containerID="40c376e0857a86587d9206e5fe8aed3604eb35f329dda3ac2e9c614551127168"
Sep 11 16:04:32.231250 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:04:32.231211    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-6vx4t" event={"ID":"a8658a10-11a7-4fda-99bb-dcd369d5918c","Type":"ContainerStarted","Data":"b6c7df6330c5b457c1af80c09c4b3a5fbe202927cbf27d575b33ad72d067e4bc"}
Sep 11 16:07:08.408976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:08.408944    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-95vwx"]
Sep 11 16:07:08.411973 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:08.411954    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-95vwx"
Sep 11 16:07:08.415277 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:08.415249    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-6strz\""
Sep 11 16:07:08.415476 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:08.415458    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 16:07:08.416196 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:08.416180    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 16:07:08.422159 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:08.422133    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-95vwx"]
Sep 11 16:07:08.539838 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:08.539800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgz4w\" (UniqueName: \"kubernetes.io/projected/bf82e103-660c-473d-8d97-df85cfe2e224-kube-api-access-lgz4w\") pod \"cert-manager-cainjector-559cdd8c87-95vwx\" (UID: \"bf82e103-660c-473d-8d97-df85cfe2e224\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-95vwx"
Sep 11 16:07:08.540018 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:08.539853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf82e103-660c-473d-8d97-df85cfe2e224-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-95vwx\" (UID: \"bf82e103-660c-473d-8d97-df85cfe2e224\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-95vwx"
Sep 11 16:07:08.640696 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:08.640656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgz4w\" (UniqueName: \"kubernetes.io/projected/bf82e103-660c-473d-8d97-df85cfe2e224-kube-api-access-lgz4w\") pod \"cert-manager-cainjector-559cdd8c87-95vwx\" (UID: \"bf82e103-660c-473d-8d97-df85cfe2e224\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-95vwx"
Sep 11 16:07:08.640872 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:08.640717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf82e103-660c-473d-8d97-df85cfe2e224-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-95vwx\" (UID: \"bf82e103-660c-473d-8d97-df85cfe2e224\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-95vwx"
Sep 11 16:07:08.650225 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:08.650199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgz4w\" (UniqueName: \"kubernetes.io/projected/bf82e103-660c-473d-8d97-df85cfe2e224-kube-api-access-lgz4w\") pod \"cert-manager-cainjector-559cdd8c87-95vwx\" (UID: \"bf82e103-660c-473d-8d97-df85cfe2e224\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-95vwx"
Sep 11 16:07:08.653335 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:08.653303    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf82e103-660c-473d-8d97-df85cfe2e224-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-95vwx\" (UID: \"bf82e103-660c-473d-8d97-df85cfe2e224\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-95vwx"
Sep 11 16:07:08.721014 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:08.720914    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-95vwx"
Sep 11 16:07:08.990872 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:08.990766    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-95vwx"]
Sep 11 16:07:08.993477 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:07:08.993451    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf82e103_660c_473d_8d97_df85cfe2e224.slice/crio-cdbaf60f7aeb72485c844edba9598761f89a29606f8860d034644804c58ba2ac WatchSource:0}: Error finding container cdbaf60f7aeb72485c844edba9598761f89a29606f8860d034644804c58ba2ac: Status 404 returned error can't find the container with id cdbaf60f7aeb72485c844edba9598761f89a29606f8860d034644804c58ba2ac
Sep 11 16:07:09.763553 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:09.763516    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-95vwx" event={"ID":"bf82e103-660c-473d-8d97-df85cfe2e224","Type":"ContainerStarted","Data":"cdbaf60f7aeb72485c844edba9598761f89a29606f8860d034644804c58ba2ac"}
Sep 11 16:07:12.777312 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:12.777277    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-95vwx" event={"ID":"bf82e103-660c-473d-8d97-df85cfe2e224","Type":"ContainerStarted","Data":"96fd054875057517360bba8823ab715aeee8b6a5c2d088ff7733e7488b28f7c4"}
Sep 11 16:07:12.804486 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:12.804433    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-95vwx" podStartSLOduration=1.752592728 podStartE2EDuration="4.804417901s" totalImagesPullingTime="3.051825173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:07:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:07:12.804195417 +0000 UTC m=+292.649838975" watchObservedRunningTime="2026-09-11 16:07:12.804417901 +0000 UTC m=+292.650061455"
Sep 11 16:07:20.504286 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:20.504260    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/1.log"
Sep 11 16:07:20.506938 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:20.506915    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/1.log"
Sep 11 16:07:20.511747 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:20.511726    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ng2ww_07d57c04-0697-419b-9d3e-ca11761da85c/ovn-acl-logging/0.log"
Sep 11 16:07:20.515028 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:20.515009    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 16:07:20.515516 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:20.515499    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ng2ww_07d57c04-0697-419b-9d3e-ca11761da85c/ovn-acl-logging/0.log"
Sep 11 16:07:36.696492 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.696459    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"]
Sep 11 16:07:36.699696 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.699682    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:07:36.703542 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.703505    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 16:07:36.703689 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.703546    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 16:07:36.703689 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.703551    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rngc9\""
Sep 11 16:07:36.703689 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.703551    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 16:07:36.703870 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.703854    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 16:07:36.728064 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.728022    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"]
Sep 11 16:07:36.739019 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.738991    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8929d82-a6e4-4e3c-a9e5-8158a14652bb-manifests\") pod \"opendatahub-operator-controller-manager-58649dcd54-t5vqz\" (UID: \"b8929d82-a6e4-4e3c-a9e5-8158a14652bb\") " pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:07:36.739019 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.739023    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8929d82-a6e4-4e3c-a9e5-8158a14652bb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58649dcd54-t5vqz\" (UID: \"b8929d82-a6e4-4e3c-a9e5-8158a14652bb\") " pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:07:36.739198 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.739060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnq87\" (UniqueName: \"kubernetes.io/projected/b8929d82-a6e4-4e3c-a9e5-8158a14652bb-kube-api-access-wnq87\") pod \"opendatahub-operator-controller-manager-58649dcd54-t5vqz\" (UID: \"b8929d82-a6e4-4e3c-a9e5-8158a14652bb\") " pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:07:36.739198 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.739104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8929d82-a6e4-4e3c-a9e5-8158a14652bb-webhook-cert\") pod \"opendatahub-operator-controller-manager-58649dcd54-t5vqz\" (UID: \"b8929d82-a6e4-4e3c-a9e5-8158a14652bb\") " pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:07:36.739198 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.739123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8929d82-a6e4-4e3c-a9e5-8158a14652bb-tmp\") pod \"opendatahub-operator-controller-manager-58649dcd54-t5vqz\" (UID: \"b8929d82-a6e4-4e3c-a9e5-8158a14652bb\") " pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:07:36.840086 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.840051    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8929d82-a6e4-4e3c-a9e5-8158a14652bb-manifests\") pod \"opendatahub-operator-controller-manager-58649dcd54-t5vqz\" (UID: \"b8929d82-a6e4-4e3c-a9e5-8158a14652bb\") " pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:07:36.840086 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.840083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8929d82-a6e4-4e3c-a9e5-8158a14652bb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58649dcd54-t5vqz\" (UID: \"b8929d82-a6e4-4e3c-a9e5-8158a14652bb\") " pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:07:36.840318 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.840102    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnq87\" (UniqueName: \"kubernetes.io/projected/b8929d82-a6e4-4e3c-a9e5-8158a14652bb-kube-api-access-wnq87\") pod \"opendatahub-operator-controller-manager-58649dcd54-t5vqz\" (UID: \"b8929d82-a6e4-4e3c-a9e5-8158a14652bb\") " pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:07:36.840318 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.840139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8929d82-a6e4-4e3c-a9e5-8158a14652bb-webhook-cert\") pod \"opendatahub-operator-controller-manager-58649dcd54-t5vqz\" (UID: \"b8929d82-a6e4-4e3c-a9e5-8158a14652bb\") " pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:07:36.840318 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.840166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8929d82-a6e4-4e3c-a9e5-8158a14652bb-tmp\") pod \"opendatahub-operator-controller-manager-58649dcd54-t5vqz\" (UID: \"b8929d82-a6e4-4e3c-a9e5-8158a14652bb\") " pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:07:36.840508 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.840487    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8929d82-a6e4-4e3c-a9e5-8158a14652bb-manifests\") pod \"opendatahub-operator-controller-manager-58649dcd54-t5vqz\" (UID: \"b8929d82-a6e4-4e3c-a9e5-8158a14652bb\") " pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:07:36.842629 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.842598    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8929d82-a6e4-4e3c-a9e5-8158a14652bb-tmp\") pod \"opendatahub-operator-controller-manager-58649dcd54-t5vqz\" (UID: \"b8929d82-a6e4-4e3c-a9e5-8158a14652bb\") " pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:07:36.842890 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.842871    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8929d82-a6e4-4e3c-a9e5-8158a14652bb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58649dcd54-t5vqz\" (UID: \"b8929d82-a6e4-4e3c-a9e5-8158a14652bb\") " pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:07:36.842968 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.842876    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8929d82-a6e4-4e3c-a9e5-8158a14652bb-webhook-cert\") pod \"opendatahub-operator-controller-manager-58649dcd54-t5vqz\" (UID: \"b8929d82-a6e4-4e3c-a9e5-8158a14652bb\") " pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:07:36.855407 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:36.855381    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnq87\" (UniqueName: \"kubernetes.io/projected/b8929d82-a6e4-4e3c-a9e5-8158a14652bb-kube-api-access-wnq87\") pod \"opendatahub-operator-controller-manager-58649dcd54-t5vqz\" (UID: \"b8929d82-a6e4-4e3c-a9e5-8158a14652bb\") " pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:07:37.009878 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:37.009774    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:07:37.294665 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:37.294642    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"]
Sep 11 16:07:37.296765 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:07:37.296735    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8929d82_a6e4_4e3c_a9e5_8158a14652bb.slice/crio-463b257b42f32d25fd3744e7f77c159866c81deebd8d44cf3d005f6c70d85578 WatchSource:0}: Error finding container 463b257b42f32d25fd3744e7f77c159866c81deebd8d44cf3d005f6c70d85578: Status 404 returned error can't find the container with id 463b257b42f32d25fd3744e7f77c159866c81deebd8d44cf3d005f6c70d85578
Sep 11 16:07:37.298506 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:37.298487    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:07:37.870791 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:37.870731    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz" event={"ID":"b8929d82-a6e4-4e3c-a9e5-8158a14652bb","Type":"ContainerStarted","Data":"463b257b42f32d25fd3744e7f77c159866c81deebd8d44cf3d005f6c70d85578"}
Sep 11 16:07:40.885532 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:40.885494    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz" event={"ID":"b8929d82-a6e4-4e3c-a9e5-8158a14652bb","Type":"ContainerStarted","Data":"af8f2c574e695fdfed670e7364071ebf92a0b82f620e19e75e557c07f1e91994"}
Sep 11 16:07:41.891702 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:41.891672    2577 generic.go:356] "Generic (PLEG): container finished" podID="b8929d82-a6e4-4e3c-a9e5-8158a14652bb" containerID="af8f2c574e695fdfed670e7364071ebf92a0b82f620e19e75e557c07f1e91994" exitCode=0
Sep 11 16:07:41.892104 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:41.891712    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz" event={"ID":"b8929d82-a6e4-4e3c-a9e5-8158a14652bb","Type":"ContainerDied","Data":"af8f2c574e695fdfed670e7364071ebf92a0b82f620e19e75e557c07f1e91994"}
Sep 11 16:07:42.896988 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:42.896948    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz" event={"ID":"b8929d82-a6e4-4e3c-a9e5-8158a14652bb","Type":"ContainerStarted","Data":"0a680931a955b912451051d8b55a5fbf25f4203294cdfdd8a5fde767d01aba7f"}
Sep 11 16:07:42.897415 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:42.897083    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:07:42.941811 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:42.941746    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz" podStartSLOduration=2.169132863 podStartE2EDuration="6.941725018s" totalImagesPullingTime="3.770070268s" totalInitContainerRuntime="1.002521887s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:07:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:07:42.940904944 +0000 UTC m=+322.786548509" watchObservedRunningTime="2026-09-11 16:07:42.941725018 +0000 UTC m=+322.787368574"
Sep 11 16:07:53.902790 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:53.902755    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:07:56.973166 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:56.973136    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5864499db4-xmxgc"]
Sep 11 16:07:56.982536 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:56.982510    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5864499db4-xmxgc"
Sep 11 16:07:56.986678 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:56.986657    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 16:07:56.987052 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:56.987019    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 16:07:56.987147 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:56.987027    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 16:07:56.987204 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:56.987139    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 16:07:56.995693 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:56.995668    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5864499db4-xmxgc"]
Sep 11 16:07:57.088065 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:57.088008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bf2e4aa-7611-4ae4-b9d2-16c18d838d14-tls-certs\") pod \"kube-auth-proxy-5864499db4-xmxgc\" (UID: \"0bf2e4aa-7611-4ae4-b9d2-16c18d838d14\") " pod="openshift-ingress/kube-auth-proxy-5864499db4-xmxgc"
Sep 11 16:07:57.088235 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:57.088083    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrhcd\" (UniqueName: \"kubernetes.io/projected/0bf2e4aa-7611-4ae4-b9d2-16c18d838d14-kube-api-access-vrhcd\") pod \"kube-auth-proxy-5864499db4-xmxgc\" (UID: \"0bf2e4aa-7611-4ae4-b9d2-16c18d838d14\") " pod="openshift-ingress/kube-auth-proxy-5864499db4-xmxgc"
Sep 11 16:07:57.088235 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:57.088121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bf2e4aa-7611-4ae4-b9d2-16c18d838d14-tmp\") pod \"kube-auth-proxy-5864499db4-xmxgc\" (UID: \"0bf2e4aa-7611-4ae4-b9d2-16c18d838d14\") " pod="openshift-ingress/kube-auth-proxy-5864499db4-xmxgc"
Sep 11 16:07:57.189069 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:57.189018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bf2e4aa-7611-4ae4-b9d2-16c18d838d14-tls-certs\") pod \"kube-auth-proxy-5864499db4-xmxgc\" (UID: \"0bf2e4aa-7611-4ae4-b9d2-16c18d838d14\") " pod="openshift-ingress/kube-auth-proxy-5864499db4-xmxgc"
Sep 11 16:07:57.189275 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:57.189080    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrhcd\" (UniqueName: \"kubernetes.io/projected/0bf2e4aa-7611-4ae4-b9d2-16c18d838d14-kube-api-access-vrhcd\") pod \"kube-auth-proxy-5864499db4-xmxgc\" (UID: \"0bf2e4aa-7611-4ae4-b9d2-16c18d838d14\") " pod="openshift-ingress/kube-auth-proxy-5864499db4-xmxgc"
Sep 11 16:07:57.189275 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:57.189108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bf2e4aa-7611-4ae4-b9d2-16c18d838d14-tmp\") pod \"kube-auth-proxy-5864499db4-xmxgc\" (UID: \"0bf2e4aa-7611-4ae4-b9d2-16c18d838d14\") " pod="openshift-ingress/kube-auth-proxy-5864499db4-xmxgc"
Sep 11 16:07:57.191679 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:57.191651    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bf2e4aa-7611-4ae4-b9d2-16c18d838d14-tmp\") pod \"kube-auth-proxy-5864499db4-xmxgc\" (UID: \"0bf2e4aa-7611-4ae4-b9d2-16c18d838d14\") " pod="openshift-ingress/kube-auth-proxy-5864499db4-xmxgc"
Sep 11 16:07:57.191935 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:57.191914    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bf2e4aa-7611-4ae4-b9d2-16c18d838d14-tls-certs\") pod \"kube-auth-proxy-5864499db4-xmxgc\" (UID: \"0bf2e4aa-7611-4ae4-b9d2-16c18d838d14\") " pod="openshift-ingress/kube-auth-proxy-5864499db4-xmxgc"
Sep 11 16:07:57.200175 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:57.200150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrhcd\" (UniqueName: \"kubernetes.io/projected/0bf2e4aa-7611-4ae4-b9d2-16c18d838d14-kube-api-access-vrhcd\") pod \"kube-auth-proxy-5864499db4-xmxgc\" (UID: \"0bf2e4aa-7611-4ae4-b9d2-16c18d838d14\") " pod="openshift-ingress/kube-auth-proxy-5864499db4-xmxgc"
Sep 11 16:07:57.294276 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:57.294198    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5864499db4-xmxgc"
Sep 11 16:07:57.570813 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:57.570740    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5864499db4-xmxgc"]
Sep 11 16:07:57.573889 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:07:57.573864    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf2e4aa_7611_4ae4_b9d2_16c18d838d14.slice/crio-5323ce06f7bed598f8b2937d895b22d779a31cfad9a37da6202b21d92ebfcde2 WatchSource:0}: Error finding container 5323ce06f7bed598f8b2937d895b22d779a31cfad9a37da6202b21d92ebfcde2: Status 404 returned error can't find the container with id 5323ce06f7bed598f8b2937d895b22d779a31cfad9a37da6202b21d92ebfcde2
Sep 11 16:07:57.949958 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:57.949923    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5864499db4-xmxgc" event={"ID":"0bf2e4aa-7611-4ae4-b9d2-16c18d838d14","Type":"ContainerStarted","Data":"5323ce06f7bed598f8b2937d895b22d779a31cfad9a37da6202b21d92ebfcde2"}
Sep 11 16:07:58.903207 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:58.903170    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"]
Sep 11 16:07:58.907238 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:58.907197    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:58.910393 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:58.910370    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 16:07:58.910393 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:58.910390    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 16:07:58.910598 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:58.910372    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 16:07:58.910598 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:58.910578    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 16:07:58.910941 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:58.910923    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-cvrlv\""
Sep 11 16:07:58.934577 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:58.934541    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"]
Sep 11 16:07:59.005022 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.004983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.005194 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.005058    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.005194 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.005097    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.005323 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.005188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.005323 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.005251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.005323 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.005296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.005455 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.005325    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wbd5\" (UniqueName: \"kubernetes.io/projected/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-kube-api-access-5wbd5\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.106118 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.106013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.106118 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.106087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wbd5\" (UniqueName: \"kubernetes.io/projected/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-kube-api-access-5wbd5\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.106380 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.106145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.106380 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.106175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.106380 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.106203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.106380 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.106240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.106380 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.106278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.107225 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.107169    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.109376 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.109325    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.110028 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.109834    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.110028 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.109890    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.110255 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.110234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.116593 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.116560    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wbd5\" (UniqueName: \"kubernetes.io/projected/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-kube-api-access-5wbd5\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.117867 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.117824    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-szbnw\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.219116 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.219006    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:07:59.518494 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:07:59.518406    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"]
Sep 11 16:08:00.620204 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:08:00.620170    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cefbadf_c2c1_46c7_b7f9_acf5f83c0e5c.slice/crio-4ce9592bed895a132cbc7f124e622813b5b24be17c12d4f5df14e8e09d6b52b0 WatchSource:0}: Error finding container 4ce9592bed895a132cbc7f124e622813b5b24be17c12d4f5df14e8e09d6b52b0: Status 404 returned error can't find the container with id 4ce9592bed895a132cbc7f124e622813b5b24be17c12d4f5df14e8e09d6b52b0
Sep 11 16:08:00.963301 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:00.963192    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5864499db4-xmxgc" event={"ID":"0bf2e4aa-7611-4ae4-b9d2-16c18d838d14","Type":"ContainerStarted","Data":"9374d5c418507a9bded376f1d249bb7fb2bde07b5533447b1cf5ed6682d656a2"}
Sep 11 16:08:00.964575 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:00.964550    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw" event={"ID":"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c","Type":"ContainerStarted","Data":"4ce9592bed895a132cbc7f124e622813b5b24be17c12d4f5df14e8e09d6b52b0"}
Sep 11 16:08:00.981308 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:00.981253    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5864499db4-xmxgc" podStartSLOduration=1.878168059 podStartE2EDuration="4.981236428s" totalImagesPullingTime="3.103068369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:08:00.97970711 +0000 UTC m=+340.825350664" watchObservedRunningTime="2026-09-11 16:08:00.981236428 +0000 UTC m=+340.826879982"
Sep 11 16:08:03.286839 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:03.286803    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 16:08:03.287116 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:03.286871    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 16:08:03.980776 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:03.980279    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw" event={"ID":"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c","Type":"ContainerStarted","Data":"3b010aeffba0dbc97d434c8caeaf302b2e15284a3897b21cec1627077b18eef4"}
Sep 11 16:08:03.980776 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:03.980565    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:08:04.009830 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:04.009777    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw" podStartSLOduration=3.345307931 podStartE2EDuration="6.009762769s" totalImagesPullingTime="2.664454838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:08:04.008635136 +0000 UTC m=+343.854278688" watchObservedRunningTime="2026-09-11 16:08:04.009762769 +0000 UTC m=+343.855406324"
Sep 11 16:08:04.985984 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:04.985951    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:08:07.674304 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:07.674259    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vncpm"]
Sep 11 16:08:07.678663 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:07.678644    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-vncpm"
Sep 11 16:08:07.681363 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:07.681338    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 11 16:08:07.681466 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:07.681421    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-5dtgq\""
Sep 11 16:08:07.695056 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:07.691142    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vncpm"]
Sep 11 16:08:07.777274 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:07.777234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47a55b39-a458-4484-af8f-07a70f9de757-cert\") pod \"odh-model-controller-7998c67dc4-vncpm\" (UID: \"47a55b39-a458-4484-af8f-07a70f9de757\") " pod="opendatahub/odh-model-controller-7998c67dc4-vncpm"
Sep 11 16:08:07.777488 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:07.777284    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jg4v\" (UniqueName: \"kubernetes.io/projected/47a55b39-a458-4484-af8f-07a70f9de757-kube-api-access-2jg4v\") pod \"odh-model-controller-7998c67dc4-vncpm\" (UID: \"47a55b39-a458-4484-af8f-07a70f9de757\") " pod="opendatahub/odh-model-controller-7998c67dc4-vncpm"
Sep 11 16:08:07.878055 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:07.877996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47a55b39-a458-4484-af8f-07a70f9de757-cert\") pod \"odh-model-controller-7998c67dc4-vncpm\" (UID: \"47a55b39-a458-4484-af8f-07a70f9de757\") " pod="opendatahub/odh-model-controller-7998c67dc4-vncpm"
Sep 11 16:08:07.878229 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:07.878067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jg4v\" (UniqueName: \"kubernetes.io/projected/47a55b39-a458-4484-af8f-07a70f9de757-kube-api-access-2jg4v\") pod \"odh-model-controller-7998c67dc4-vncpm\" (UID: \"47a55b39-a458-4484-af8f-07a70f9de757\") " pod="opendatahub/odh-model-controller-7998c67dc4-vncpm"
Sep 11 16:08:07.880521 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:07.880499    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47a55b39-a458-4484-af8f-07a70f9de757-cert\") pod \"odh-model-controller-7998c67dc4-vncpm\" (UID: \"47a55b39-a458-4484-af8f-07a70f9de757\") " pod="opendatahub/odh-model-controller-7998c67dc4-vncpm"
Sep 11 16:08:07.886951 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:07.886926    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jg4v\" (UniqueName: \"kubernetes.io/projected/47a55b39-a458-4484-af8f-07a70f9de757-kube-api-access-2jg4v\") pod \"odh-model-controller-7998c67dc4-vncpm\" (UID: \"47a55b39-a458-4484-af8f-07a70f9de757\") " pod="opendatahub/odh-model-controller-7998c67dc4-vncpm"
Sep 11 16:08:07.989690 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:07.989590    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-vncpm"
Sep 11 16:08:08.210499 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:08.210466    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nzzfd"]
Sep 11 16:08:08.215147 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:08.215123    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-nzzfd"
Sep 11 16:08:08.218373 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:08.218341    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 11 16:08:08.225914 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:08.225887    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nzzfd"]
Sep 11 16:08:08.278388 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:08.278366    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vncpm"]
Sep 11 16:08:08.280676 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:08:08.280648    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a55b39_a458_4484_af8f_07a70f9de757.slice/crio-9dd7896b6ed61f958ba443851f4e6d06d2e4938af3c26362d4381bb5fb0558b1 WatchSource:0}: Error finding container 9dd7896b6ed61f958ba443851f4e6d06d2e4938af3c26362d4381bb5fb0558b1: Status 404 returned error can't find the container with id 9dd7896b6ed61f958ba443851f4e6d06d2e4938af3c26362d4381bb5fb0558b1
Sep 11 16:08:08.281921 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:08.281802    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b17b5a-de65-4253-8a1b-de2adb5ae416-tls-certs\") pod \"model-serving-api-54d89bd5fc-nzzfd\" (UID: \"65b17b5a-de65-4253-8a1b-de2adb5ae416\") " pod="opendatahub/model-serving-api-54d89bd5fc-nzzfd"
Sep 11 16:08:08.281921 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:08.281872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcwsc\" (UniqueName: \"kubernetes.io/projected/65b17b5a-de65-4253-8a1b-de2adb5ae416-kube-api-access-wcwsc\") pod \"model-serving-api-54d89bd5fc-nzzfd\" (UID: \"65b17b5a-de65-4253-8a1b-de2adb5ae416\") " pod="opendatahub/model-serving-api-54d89bd5fc-nzzfd"
Sep 11 16:08:08.382998 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:08.382963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcwsc\" (UniqueName: \"kubernetes.io/projected/65b17b5a-de65-4253-8a1b-de2adb5ae416-kube-api-access-wcwsc\") pod \"model-serving-api-54d89bd5fc-nzzfd\" (UID: \"65b17b5a-de65-4253-8a1b-de2adb5ae416\") " pod="opendatahub/model-serving-api-54d89bd5fc-nzzfd"
Sep 11 16:08:08.383193 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:08.383086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b17b5a-de65-4253-8a1b-de2adb5ae416-tls-certs\") pod \"model-serving-api-54d89bd5fc-nzzfd\" (UID: \"65b17b5a-de65-4253-8a1b-de2adb5ae416\") " pod="opendatahub/model-serving-api-54d89bd5fc-nzzfd"
Sep 11 16:08:08.383260 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:08:08.383204    2577 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 11 16:08:08.383350 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:08:08.383266    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65b17b5a-de65-4253-8a1b-de2adb5ae416-tls-certs podName:65b17b5a-de65-4253-8a1b-de2adb5ae416 nodeName:}" failed. No retries permitted until 2026-09-11 16:08:08.883246142 +0000 UTC m=+348.728889682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/65b17b5a-de65-4253-8a1b-de2adb5ae416-tls-certs") pod "model-serving-api-54d89bd5fc-nzzfd" (UID: "65b17b5a-de65-4253-8a1b-de2adb5ae416") : secret "model-serving-api-tls" not found
Sep 11 16:08:08.393501 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:08.393471    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcwsc\" (UniqueName: \"kubernetes.io/projected/65b17b5a-de65-4253-8a1b-de2adb5ae416-kube-api-access-wcwsc\") pod \"model-serving-api-54d89bd5fc-nzzfd\" (UID: \"65b17b5a-de65-4253-8a1b-de2adb5ae416\") " pod="opendatahub/model-serving-api-54d89bd5fc-nzzfd"
Sep 11 16:08:08.887027 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:08.886992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b17b5a-de65-4253-8a1b-de2adb5ae416-tls-certs\") pod \"model-serving-api-54d89bd5fc-nzzfd\" (UID: \"65b17b5a-de65-4253-8a1b-de2adb5ae416\") " pod="opendatahub/model-serving-api-54d89bd5fc-nzzfd"
Sep 11 16:08:08.889532 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:08.889513    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b17b5a-de65-4253-8a1b-de2adb5ae416-tls-certs\") pod \"model-serving-api-54d89bd5fc-nzzfd\" (UID: \"65b17b5a-de65-4253-8a1b-de2adb5ae416\") " pod="opendatahub/model-serving-api-54d89bd5fc-nzzfd"
Sep 11 16:08:09.002903 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:09.002870    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-vncpm" event={"ID":"47a55b39-a458-4484-af8f-07a70f9de757","Type":"ContainerStarted","Data":"9dd7896b6ed61f958ba443851f4e6d06d2e4938af3c26362d4381bb5fb0558b1"}
Sep 11 16:08:09.129684 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:09.129643    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-nzzfd"
Sep 11 16:08:09.420350 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:09.420320    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nzzfd"]
Sep 11 16:08:09.423730 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:08:09.423695    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b17b5a_de65_4253_8a1b_de2adb5ae416.slice/crio-c868aba91f5b24b6b13b39f5ed3bc4153c9fe49a6cb1db72e13244aa5090f6a8 WatchSource:0}: Error finding container c868aba91f5b24b6b13b39f5ed3bc4153c9fe49a6cb1db72e13244aa5090f6a8: Status 404 returned error can't find the container with id c868aba91f5b24b6b13b39f5ed3bc4153c9fe49a6cb1db72e13244aa5090f6a8
Sep 11 16:08:10.008347 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:10.008310    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-nzzfd" event={"ID":"65b17b5a-de65-4253-8a1b-de2adb5ae416","Type":"ContainerStarted","Data":"c868aba91f5b24b6b13b39f5ed3bc4153c9fe49a6cb1db72e13244aa5090f6a8"}
Sep 11 16:08:13.024191 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:13.024148    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-nzzfd" event={"ID":"65b17b5a-de65-4253-8a1b-de2adb5ae416","Type":"ContainerStarted","Data":"4d65c46bb57215fe7d9616df0e8f830000c48d84b74307d4b39b0112868e436c"}
Sep 11 16:08:13.024702 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:13.024278    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-nzzfd"
Sep 11 16:08:13.025814 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:13.025791    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-vncpm" event={"ID":"47a55b39-a458-4484-af8f-07a70f9de757","Type":"ContainerStarted","Data":"4694e8a00e39848e9b8e750976de1c0234980eed356773926d574216a4ff4ae4"}
Sep 11 16:08:13.025949 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:13.025931    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-vncpm"
Sep 11 16:08:13.045060 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:13.044985    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-nzzfd" podStartSLOduration=2.475558096 podStartE2EDuration="5.044970495s" totalImagesPullingTime="2.569412399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:08:13.043631326 +0000 UTC m=+352.889274882" watchObservedRunningTime="2026-09-11 16:08:13.044970495 +0000 UTC m=+352.890614050"
Sep 11 16:08:13.060507 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:13.060456    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-vncpm" podStartSLOduration=2.344914146 podStartE2EDuration="6.060440671s" totalImagesPullingTime="3.715526525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:08:13.059715392 +0000 UTC m=+352.905358947" watchObservedRunningTime="2026-09-11 16:08:13.060440671 +0000 UTC m=+352.906084227"
Sep 11 16:08:18.357673 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.357636    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp"]
Sep 11 16:08:18.362273 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.362256    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp"
Sep 11 16:08:18.366070 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.366002    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 11 16:08:18.366070 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.366018    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77"]
Sep 11 16:08:18.366765 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.366739    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-ppcbw\""
Sep 11 16:08:18.368339 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.368319    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77"
Sep 11 16:08:18.371319 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.371299    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-j2wnm\""
Sep 11 16:08:18.371415 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.371400    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 11 16:08:18.394209 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.394176    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp"]
Sep 11 16:08:18.395555 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.395532    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77"]
Sep 11 16:08:18.468014 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.467977    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e4cb213-1072-4f8d-9199-9f4c0476c454-cert\") pod \"kserve-controller-manager-d6f4b65b8-2kmcp\" (UID: \"3e4cb213-1072-4f8d-9199-9f4c0476c454\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp"
Sep 11 16:08:18.468205 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.468056    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47gh\" (UniqueName: \"kubernetes.io/projected/3e4cb213-1072-4f8d-9199-9f4c0476c454-kube-api-access-x47gh\") pod \"kserve-controller-manager-d6f4b65b8-2kmcp\" (UID: \"3e4cb213-1072-4f8d-9199-9f4c0476c454\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp"
Sep 11 16:08:18.468205 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.468087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j969g\" (UniqueName: \"kubernetes.io/projected/415703dd-3c9d-487b-a384-2dee750c0915-kube-api-access-j969g\") pod \"llmisvc-controller-manager-777c6bd7d5-hcm77\" (UID: \"415703dd-3c9d-487b-a384-2dee750c0915\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77"
Sep 11 16:08:18.468205 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.468168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/415703dd-3c9d-487b-a384-2dee750c0915-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hcm77\" (UID: \"415703dd-3c9d-487b-a384-2dee750c0915\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77"
Sep 11 16:08:18.569348 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.569303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47gh\" (UniqueName: \"kubernetes.io/projected/3e4cb213-1072-4f8d-9199-9f4c0476c454-kube-api-access-x47gh\") pod \"kserve-controller-manager-d6f4b65b8-2kmcp\" (UID: \"3e4cb213-1072-4f8d-9199-9f4c0476c454\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp"
Sep 11 16:08:18.569348 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.569350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j969g\" (UniqueName: \"kubernetes.io/projected/415703dd-3c9d-487b-a384-2dee750c0915-kube-api-access-j969g\") pod \"llmisvc-controller-manager-777c6bd7d5-hcm77\" (UID: \"415703dd-3c9d-487b-a384-2dee750c0915\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77"
Sep 11 16:08:18.569618 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.569397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/415703dd-3c9d-487b-a384-2dee750c0915-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hcm77\" (UID: \"415703dd-3c9d-487b-a384-2dee750c0915\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77"
Sep 11 16:08:18.569618 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.569437    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e4cb213-1072-4f8d-9199-9f4c0476c454-cert\") pod \"kserve-controller-manager-d6f4b65b8-2kmcp\" (UID: \"3e4cb213-1072-4f8d-9199-9f4c0476c454\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp"
Sep 11 16:08:18.572086 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.572054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e4cb213-1072-4f8d-9199-9f4c0476c454-cert\") pod \"kserve-controller-manager-d6f4b65b8-2kmcp\" (UID: \"3e4cb213-1072-4f8d-9199-9f4c0476c454\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp"
Sep 11 16:08:18.572086 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.572085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/415703dd-3c9d-487b-a384-2dee750c0915-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hcm77\" (UID: \"415703dd-3c9d-487b-a384-2dee750c0915\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77"
Sep 11 16:08:18.579471 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.579447    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47gh\" (UniqueName: \"kubernetes.io/projected/3e4cb213-1072-4f8d-9199-9f4c0476c454-kube-api-access-x47gh\") pod \"kserve-controller-manager-d6f4b65b8-2kmcp\" (UID: \"3e4cb213-1072-4f8d-9199-9f4c0476c454\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp"
Sep 11 16:08:18.579598 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.579581    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j969g\" (UniqueName: \"kubernetes.io/projected/415703dd-3c9d-487b-a384-2dee750c0915-kube-api-access-j969g\") pod \"llmisvc-controller-manager-777c6bd7d5-hcm77\" (UID: \"415703dd-3c9d-487b-a384-2dee750c0915\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77"
Sep 11 16:08:18.674782 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.674690    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp"
Sep 11 16:08:18.680768 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.680746    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77"
Sep 11 16:08:18.963258 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.963178    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp"]
Sep 11 16:08:18.966418 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:08:18.966390    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e4cb213_1072_4f8d_9199_9f4c0476c454.slice/crio-fb38bd2dde95f6713dc83f3486864bc998720181d4e06277a5856db95b3e1b24 WatchSource:0}: Error finding container fb38bd2dde95f6713dc83f3486864bc998720181d4e06277a5856db95b3e1b24: Status 404 returned error can't find the container with id fb38bd2dde95f6713dc83f3486864bc998720181d4e06277a5856db95b3e1b24
Sep 11 16:08:18.970341 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:18.970290    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77"]
Sep 11 16:08:18.972257 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:08:18.972232    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod415703dd_3c9d_487b_a384_2dee750c0915.slice/crio-02aa7948c5a4baef51a112c6fd13373bfa415a74903aae92fcaab32659c04acb WatchSource:0}: Error finding container 02aa7948c5a4baef51a112c6fd13373bfa415a74903aae92fcaab32659c04acb: Status 404 returned error can't find the container with id 02aa7948c5a4baef51a112c6fd13373bfa415a74903aae92fcaab32659c04acb
Sep 11 16:08:19.051115 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:19.051081    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp" event={"ID":"3e4cb213-1072-4f8d-9199-9f4c0476c454","Type":"ContainerStarted","Data":"fb38bd2dde95f6713dc83f3486864bc998720181d4e06277a5856db95b3e1b24"}
Sep 11 16:08:19.052269 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:19.052244    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77" event={"ID":"415703dd-3c9d-487b-a384-2dee750c0915","Type":"ContainerStarted","Data":"02aa7948c5a4baef51a112c6fd13373bfa415a74903aae92fcaab32659c04acb"}
Sep 11 16:08:19.578575 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:19.578494    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"]
Sep 11 16:08:19.579206 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:19.578746    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw" podUID="9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c" containerName="discovery" containerID="cri-o://3b010aeffba0dbc97d434c8caeaf302b2e15284a3897b21cec1627077b18eef4" gracePeriod=30
Sep 11 16:08:19.954451 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:19.954409    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:08:20.059419 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.059383    2577 generic.go:356] "Generic (PLEG): container finished" podID="9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c" containerID="3b010aeffba0dbc97d434c8caeaf302b2e15284a3897b21cec1627077b18eef4" exitCode=0
Sep 11 16:08:20.059590 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.059435    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw" event={"ID":"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c","Type":"ContainerDied","Data":"3b010aeffba0dbc97d434c8caeaf302b2e15284a3897b21cec1627077b18eef4"}
Sep 11 16:08:20.059590 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.059479    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw" event={"ID":"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c","Type":"ContainerDied","Data":"4ce9592bed895a132cbc7f124e622813b5b24be17c12d4f5df14e8e09d6b52b0"}
Sep 11 16:08:20.059590 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.059504    2577 scope.go:117] "RemoveContainer" containerID="3b010aeffba0dbc97d434c8caeaf302b2e15284a3897b21cec1627077b18eef4"
Sep 11 16:08:20.059590 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.059438    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"
Sep 11 16:08:20.081795 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.081766    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-kubeconfig\") pod \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") "
Sep 11 16:08:20.081961 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.081823    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-token\") pod \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") "
Sep 11 16:08:20.081961 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.081844    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-local-certs\") pod \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") "
Sep 11 16:08:20.081961 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.081864    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-cacerts\") pod \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") "
Sep 11 16:08:20.081961 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.081905    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-csr-ca-configmap\") pod \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") "
Sep 11 16:08:20.081961 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.081959    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-csr-dns-cert\") pod \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") "
Sep 11 16:08:20.082275 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.081981    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wbd5\" (UniqueName: \"kubernetes.io/projected/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-kube-api-access-5wbd5\") pod \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\" (UID: \"9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c\") "
Sep 11 16:08:20.082430 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.082382    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c" (UID: "9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:08:20.084617 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.084583    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-kube-api-access-5wbd5" (OuterVolumeSpecName: "kube-api-access-5wbd5") pod "9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c" (UID: "9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c"). InnerVolumeSpecName "kube-api-access-5wbd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:08:20.084747 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.084666    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c" (UID: "9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:08:20.084816 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.084783    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-token" (OuterVolumeSpecName: "istio-token") pod "9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c" (UID: "9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:08:20.084898 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.084880    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-cacerts" (OuterVolumeSpecName: "cacerts") pod "9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c" (UID: "9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:08:20.084898 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.084884    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-local-certs" (OuterVolumeSpecName: "local-certs") pod "9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c" (UID: "9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:08:20.085170 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.085149    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c" (UID: "9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:08:20.144491 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.144464    2577 scope.go:117] "RemoveContainer" containerID="3b010aeffba0dbc97d434c8caeaf302b2e15284a3897b21cec1627077b18eef4"
Sep 11 16:08:20.144909 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:08:20.144884    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b010aeffba0dbc97d434c8caeaf302b2e15284a3897b21cec1627077b18eef4\": container with ID starting with 3b010aeffba0dbc97d434c8caeaf302b2e15284a3897b21cec1627077b18eef4 not found: ID does not exist" containerID="3b010aeffba0dbc97d434c8caeaf302b2e15284a3897b21cec1627077b18eef4"
Sep 11 16:08:20.144971 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.144921    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b010aeffba0dbc97d434c8caeaf302b2e15284a3897b21cec1627077b18eef4"} err="failed to get container status \"3b010aeffba0dbc97d434c8caeaf302b2e15284a3897b21cec1627077b18eef4\": rpc error: code = NotFound desc = could not find container \"3b010aeffba0dbc97d434c8caeaf302b2e15284a3897b21cec1627077b18eef4\": container with ID starting with 3b010aeffba0dbc97d434c8caeaf302b2e15284a3897b21cec1627077b18eef4 not found: ID does not exist"
Sep 11 16:08:20.183384 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.183343    2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-kubeconfig\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:08:20.183384 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.183374    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-token\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:08:20.183384 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.183387    2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-local-certs\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:08:20.183638 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.183400    2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-cacerts\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:08:20.183638 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.183413    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-csr-ca-configmap\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:08:20.183638 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.183424    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-istio-csr-dns-cert\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:08:20.183638 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.183436    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wbd5\" (UniqueName: \"kubernetes.io/projected/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c-kube-api-access-5wbd5\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:08:20.383469 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.383432    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"]
Sep 11 16:08:20.387065 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.386984    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-szbnw"]
Sep 11 16:08:20.633782 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:20.633698    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c" path="/var/lib/kubelet/pods/9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c/volumes"
Sep 11 16:08:22.071782 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:22.071743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp" event={"ID":"3e4cb213-1072-4f8d-9199-9f4c0476c454","Type":"ContainerStarted","Data":"8e36de8a34c8b50cf29ea9266b7b6d8ba43ed514aaa828504e05604226eed7e1"}
Sep 11 16:08:22.072295 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:22.072070    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp"
Sep 11 16:08:22.098393 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:22.098332    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp" podStartSLOduration=1.784502219 podStartE2EDuration="4.098314406s" totalImagesPullingTime="2.313812187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:08:22.095777247 +0000 UTC m=+361.941420805" watchObservedRunningTime="2026-09-11 16:08:22.098314406 +0000 UTC m=+361.943957965"
Sep 11 16:08:23.077357 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:23.077311    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77" event={"ID":"415703dd-3c9d-487b-a384-2dee750c0915","Type":"ContainerStarted","Data":"4d6f5fbb5133b03bc35263afb37c02827cbed6cffa87711e11e4b856c268ccab"}
Sep 11 16:08:23.077822 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:23.077534    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77"
Sep 11 16:08:23.119605 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:23.119550    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77" podStartSLOduration=1.918026405 podStartE2EDuration="5.119533634s" totalImagesPullingTime="3.201507229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:08:23.119425167 +0000 UTC m=+362.965068723" watchObservedRunningTime="2026-09-11 16:08:23.119533634 +0000 UTC m=+362.965177188"
Sep 11 16:08:24.031645 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:24.031616    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-vncpm"
Sep 11 16:08:24.034813 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:24.034789    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-nzzfd"
Sep 11 16:08:53.083065 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:53.083005    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp"
Sep 11 16:08:54.085180 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:08:54.085150    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77"
Sep 11 16:09:34.521420 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:34.521385    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z"]
Sep 11 16:09:34.521901 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:34.521672    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c" containerName="discovery"
Sep 11 16:09:34.521901 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:34.521682    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c" containerName="discovery"
Sep 11 16:09:34.521901 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:34.521747    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cefbadf-c2c1-46c7-b7f9-acf5f83c0e5c" containerName="discovery"
Sep 11 16:09:34.524601 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:34.524569    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z"
Sep 11 16:09:34.531481 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:34.531462    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 16:09:34.532415 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:34.532397    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 16:09:34.532559 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:34.532540    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-9s665\""
Sep 11 16:09:34.533540 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:34.533525    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 16:09:34.543584 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:34.543562    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z"]
Sep 11 16:09:34.657660 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:34.657623    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd6gv\" (UniqueName: \"kubernetes.io/projected/b403831a-b82b-486e-9c63-2b7382bb238e-kube-api-access-dd6gv\") pod \"dns-operator-controller-manager-7b84cd7c9c-7p78z\" (UID: \"b403831a-b82b-486e-9c63-2b7382bb238e\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z"
Sep 11 16:09:34.758895 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:34.758855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd6gv\" (UniqueName: \"kubernetes.io/projected/b403831a-b82b-486e-9c63-2b7382bb238e-kube-api-access-dd6gv\") pod \"dns-operator-controller-manager-7b84cd7c9c-7p78z\" (UID: \"b403831a-b82b-486e-9c63-2b7382bb238e\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z"
Sep 11 16:09:34.779487 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:34.779414    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd6gv\" (UniqueName: \"kubernetes.io/projected/b403831a-b82b-486e-9c63-2b7382bb238e-kube-api-access-dd6gv\") pod \"dns-operator-controller-manager-7b84cd7c9c-7p78z\" (UID: \"b403831a-b82b-486e-9c63-2b7382bb238e\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z"
Sep 11 16:09:34.834727 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:34.834673    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z"
Sep 11 16:09:35.165392 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:35.165199    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z"]
Sep 11 16:09:35.168343 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:09:35.168309    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb403831a_b82b_486e_9c63_2b7382bb238e.slice/crio-6df387d52af7d7f0b29993a338ed5aef07671876d8889f6b77c8c6a81b4b48a6 WatchSource:0}: Error finding container 6df387d52af7d7f0b29993a338ed5aef07671876d8889f6b77c8c6a81b4b48a6: Status 404 returned error can't find the container with id 6df387d52af7d7f0b29993a338ed5aef07671876d8889f6b77c8c6a81b4b48a6
Sep 11 16:09:35.365314 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:35.365279    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z" event={"ID":"b403831a-b82b-486e-9c63-2b7382bb238e","Type":"ContainerStarted","Data":"6df387d52af7d7f0b29993a338ed5aef07671876d8889f6b77c8c6a81b4b48a6"}
Sep 11 16:09:38.381123 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:38.381083    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z" event={"ID":"b403831a-b82b-486e-9c63-2b7382bb238e","Type":"ContainerStarted","Data":"1313368003d0e8ca824db09c7d88813fce957ac7654cb84966af3dbe4aceff53"}
Sep 11 16:09:38.381589 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:38.381153    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z"
Sep 11 16:09:38.404070 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:38.403855    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z" podStartSLOduration=2.003684016 podStartE2EDuration="4.403832273s" totalImagesPullingTime="2.400148257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:38.401945688 +0000 UTC m=+438.247589246" watchObservedRunningTime="2026-09-11 16:09:38.403832273 +0000 UTC m=+438.249475829"
Sep 11 16:09:39.177152 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:39.177115    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd"]
Sep 11 16:09:39.180520 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:39.180499    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd"
Sep 11 16:09:39.184083 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:39.184056    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xhm2z\""
Sep 11 16:09:39.204618 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:39.204589    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd"]
Sep 11 16:09:39.298378 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:39.298341    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvmc5\" (UniqueName: \"kubernetes.io/projected/c0b2ff8b-a631-4917-a6a6-a2d06654aadc-kube-api-access-hvmc5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kdrfd\" (UID: \"c0b2ff8b-a631-4917-a6a6-a2d06654aadc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd"
Sep 11 16:09:39.298555 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:39.298392    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0b2ff8b-a631-4917-a6a6-a2d06654aadc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kdrfd\" (UID: \"c0b2ff8b-a631-4917-a6a6-a2d06654aadc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd"
Sep 11 16:09:39.399165 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:39.399133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvmc5\" (UniqueName: \"kubernetes.io/projected/c0b2ff8b-a631-4917-a6a6-a2d06654aadc-kube-api-access-hvmc5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kdrfd\" (UID: \"c0b2ff8b-a631-4917-a6a6-a2d06654aadc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd"
Sep 11 16:09:39.399165 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:39.399173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0b2ff8b-a631-4917-a6a6-a2d06654aadc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kdrfd\" (UID: \"c0b2ff8b-a631-4917-a6a6-a2d06654aadc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd"
Sep 11 16:09:39.399592 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:39.399487    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0b2ff8b-a631-4917-a6a6-a2d06654aadc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kdrfd\" (UID: \"c0b2ff8b-a631-4917-a6a6-a2d06654aadc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd"
Sep 11 16:09:39.421418 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:39.421386    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvmc5\" (UniqueName: \"kubernetes.io/projected/c0b2ff8b-a631-4917-a6a6-a2d06654aadc-kube-api-access-hvmc5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kdrfd\" (UID: \"c0b2ff8b-a631-4917-a6a6-a2d06654aadc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd"
Sep 11 16:09:39.490262 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:39.490177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd"
Sep 11 16:09:39.780474 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:39.780450    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd"]
Sep 11 16:09:39.782890 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:09:39.782856    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b2ff8b_a631_4917_a6a6_a2d06654aadc.slice/crio-07816edffde03f4f84791868e4ca1970513049c0b0f53ee69850f01679b3c167 WatchSource:0}: Error finding container 07816edffde03f4f84791868e4ca1970513049c0b0f53ee69850f01679b3c167: Status 404 returned error can't find the container with id 07816edffde03f4f84791868e4ca1970513049c0b0f53ee69850f01679b3c167
Sep 11 16:09:40.391646 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:40.391606    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd" event={"ID":"c0b2ff8b-a631-4917-a6a6-a2d06654aadc","Type":"ContainerStarted","Data":"07816edffde03f4f84791868e4ca1970513049c0b0f53ee69850f01679b3c167"}
Sep 11 16:09:41.948097 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:41.947998    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-zgb8c"]
Sep 11 16:09:41.951533 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:41.951505    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c"
Sep 11 16:09:41.955804 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:41.955783    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-n4kc5\""
Sep 11 16:09:41.966543 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:41.966512    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-zgb8c"]
Sep 11 16:09:42.021786 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:42.021742    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fnwt\" (UniqueName: \"kubernetes.io/projected/89912015-3e6c-41e8-8ce3-1af04df3584d-kube-api-access-4fnwt\") pod \"authorino-operator-76b478bd78-zgb8c\" (UID: \"89912015-3e6c-41e8-8ce3-1af04df3584d\") " pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c"
Sep 11 16:09:42.122302 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:42.122265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fnwt\" (UniqueName: \"kubernetes.io/projected/89912015-3e6c-41e8-8ce3-1af04df3584d-kube-api-access-4fnwt\") pod \"authorino-operator-76b478bd78-zgb8c\" (UID: \"89912015-3e6c-41e8-8ce3-1af04df3584d\") " pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c"
Sep 11 16:09:42.138997 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:42.138963    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fnwt\" (UniqueName: \"kubernetes.io/projected/89912015-3e6c-41e8-8ce3-1af04df3584d-kube-api-access-4fnwt\") pod \"authorino-operator-76b478bd78-zgb8c\" (UID: \"89912015-3e6c-41e8-8ce3-1af04df3584d\") " pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c"
Sep 11 16:09:42.264262 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:42.264174    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c"
Sep 11 16:09:43.408799 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:43.408751    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd" event={"ID":"c0b2ff8b-a631-4917-a6a6-a2d06654aadc","Type":"ContainerStarted","Data":"a5a1668a677563b224f72431adf574d84e9c0a7b762b43b12985430b102f1280"}
Sep 11 16:09:43.409294 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:43.408961    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd"
Sep 11 16:09:43.448185 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:43.448129    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd" podStartSLOduration=0.891931264 podStartE2EDuration="4.448111628s" totalImagesPullingTime="3.556180364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:43.446108539 +0000 UTC m=+443.291752090" watchObservedRunningTime="2026-09-11 16:09:43.448111628 +0000 UTC m=+443.293755182"
Sep 11 16:09:43.543659 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:43.543633    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-zgb8c"]
Sep 11 16:09:43.546238 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:09:43.546203    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89912015_3e6c_41e8_8ce3_1af04df3584d.slice/crio-df4d9b30e863a2694cd7369766a1c493976eab03199542c7b09d26030bc04a9f WatchSource:0}: Error finding container df4d9b30e863a2694cd7369766a1c493976eab03199542c7b09d26030bc04a9f: Status 404 returned error can't find the container with id df4d9b30e863a2694cd7369766a1c493976eab03199542c7b09d26030bc04a9f
Sep 11 16:09:44.034582 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:44.034545    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"]
Sep 11 16:09:44.042368 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:44.042336    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"
Sep 11 16:09:44.050247 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:44.050225    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tqngz\""
Sep 11 16:09:44.068465 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:44.068436    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"]
Sep 11 16:09:44.137097 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:44.137048    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngj5x\" (UniqueName: \"kubernetes.io/projected/c0ec6069-5ff9-4a99-9c9e-9725f11467e4-kube-api-access-ngj5x\") pod \"limitador-operator-controller-manager-74495c69f8-6j489\" (UID: \"c0ec6069-5ff9-4a99-9c9e-9725f11467e4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"
Sep 11 16:09:44.238076 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:44.238000    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngj5x\" (UniqueName: \"kubernetes.io/projected/c0ec6069-5ff9-4a99-9c9e-9725f11467e4-kube-api-access-ngj5x\") pod \"limitador-operator-controller-manager-74495c69f8-6j489\" (UID: \"c0ec6069-5ff9-4a99-9c9e-9725f11467e4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"
Sep 11 16:09:44.277667 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:44.277634    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngj5x\" (UniqueName: \"kubernetes.io/projected/c0ec6069-5ff9-4a99-9c9e-9725f11467e4-kube-api-access-ngj5x\") pod \"limitador-operator-controller-manager-74495c69f8-6j489\" (UID: \"c0ec6069-5ff9-4a99-9c9e-9725f11467e4\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"
Sep 11 16:09:44.353207 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:44.353171    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"
Sep 11 16:09:44.419546 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:44.419512    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c" event={"ID":"89912015-3e6c-41e8-8ce3-1af04df3584d","Type":"ContainerStarted","Data":"df4d9b30e863a2694cd7369766a1c493976eab03199542c7b09d26030bc04a9f"}
Sep 11 16:09:44.657719 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:44.657634    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"]
Sep 11 16:09:44.718524 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:09:44.718487    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ec6069_5ff9_4a99_9c9e_9725f11467e4.slice/crio-7b2a67baabbec184bddde510680b7b86b8f55cd90b081e50b0b683a66d7d46be WatchSource:0}: Error finding container 7b2a67baabbec184bddde510680b7b86b8f55cd90b081e50b0b683a66d7d46be: Status 404 returned error can't find the container with id 7b2a67baabbec184bddde510680b7b86b8f55cd90b081e50b0b683a66d7d46be
Sep 11 16:09:45.427507 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:45.427267    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c" event={"ID":"89912015-3e6c-41e8-8ce3-1af04df3584d","Type":"ContainerStarted","Data":"d247bb25b8df8d65d536890247acef3bf4add5dda37127dd333ba2dcd1547a02"}
Sep 11 16:09:45.427507 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:45.427370    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c"
Sep 11 16:09:45.428949 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:45.428924    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489" event={"ID":"c0ec6069-5ff9-4a99-9c9e-9725f11467e4","Type":"ContainerStarted","Data":"7b2a67baabbec184bddde510680b7b86b8f55cd90b081e50b0b683a66d7d46be"}
Sep 11 16:09:45.504843 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:45.504776    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c" podStartSLOduration=3.2761551190000002 podStartE2EDuration="4.504752289s" totalImagesPullingTime="1.22859717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:45.50369958 +0000 UTC m=+445.349343128" watchObservedRunningTime="2026-09-11 16:09:45.504752289 +0000 UTC m=+445.350395848"
Sep 11 16:09:46.434401 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:46.434366    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489" event={"ID":"c0ec6069-5ff9-4a99-9c9e-9725f11467e4","Type":"ContainerStarted","Data":"532f8986e406525f183381a9d5033c313cfc3cce6bc0f99425d3359448cf0f52"}
Sep 11 16:09:46.434825 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:46.434607    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"
Sep 11 16:09:49.387930 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:49.387903    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z"
Sep 11 16:09:49.410132 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:49.410077    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489" podStartSLOduration=5.043105777 podStartE2EDuration="6.410052341s" totalImagesPullingTime="1.366946564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:46.482492932 +0000 UTC m=+446.328136486" watchObservedRunningTime="2026-09-11 16:09:49.410052341 +0000 UTC m=+449.255695887"
Sep 11 16:09:54.421621 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:54.421589    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd"
Sep 11 16:09:56.436601 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.436571    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c"
Sep 11 16:09:56.611809 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.611762    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd"]
Sep 11 16:09:56.612078 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.612015    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd" podUID="c0b2ff8b-a631-4917-a6a6-a2d06654aadc" containerName="manager" containerID="cri-o://a5a1668a677563b224f72431adf574d84e9c0a7b762b43b12985430b102f1280" gracePeriod=2
Sep 11 16:09:56.618698 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.618669    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd"]
Sep 11 16:09:56.662947 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.662916    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"]
Sep 11 16:09:56.663179 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.663129    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489" podUID="c0ec6069-5ff9-4a99-9c9e-9725f11467e4" containerName="manager" containerID="cri-o://532f8986e406525f183381a9d5033c313cfc3cce6bc0f99425d3359448cf0f52" gracePeriod=2
Sep 11 16:09:56.665083 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.665060    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"
Sep 11 16:09:56.677348 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.677320    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"]
Sep 11 16:09:56.691700 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.691632    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh"]
Sep 11 16:09:56.692112 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.692094    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b2ff8b-a631-4917-a6a6-a2d06654aadc" containerName="manager"
Sep 11 16:09:56.692179 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.692116    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b2ff8b-a631-4917-a6a6-a2d06654aadc" containerName="manager"
Sep 11 16:09:56.692179 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.692138    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0ec6069-5ff9-4a99-9c9e-9725f11467e4" containerName="manager"
Sep 11 16:09:56.692179 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.692146    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0ec6069-5ff9-4a99-9c9e-9725f11467e4" containerName="manager"
Sep 11 16:09:56.692336 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.692236    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0ec6069-5ff9-4a99-9c9e-9725f11467e4" containerName="manager"
Sep 11 16:09:56.692336 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.692246    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0b2ff8b-a631-4917-a6a6-a2d06654aadc" containerName="manager"
Sep 11 16:09:56.694445 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:09:56.694416    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-6j489\" is forbidden: User \"system:node:ip-10-0-139-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-140.ec2.internal' and this object" podUID="c0ec6069-5ff9-4a99-9c9e-9725f11467e4" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"
Sep 11 16:09:56.695349 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.695332    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh"
Sep 11 16:09:56.699875 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:09:56.699838    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-6j489\" is forbidden: User \"system:node:ip-10-0-139-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-140.ec2.internal' and this object" podUID="c0ec6069-5ff9-4a99-9c9e-9725f11467e4" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"
Sep 11 16:09:56.736992 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.736962    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf"]
Sep 11 16:09:56.740637 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.740614    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf"
Sep 11 16:09:56.746744 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:09:56.746714    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-6j489\" is forbidden: User \"system:node:ip-10-0-139-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-140.ec2.internal' and this object" podUID="c0ec6069-5ff9-4a99-9c9e-9725f11467e4" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"
Sep 11 16:09:56.755729 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.755701    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf"]
Sep 11 16:09:56.820733 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.820693    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh"]
Sep 11 16:09:56.844280 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.844243    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmf69\" (UniqueName: \"kubernetes.io/projected/ddae13eb-94cf-4d57-8d16-0807a8c63423-kube-api-access-tmf69\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lfdnh\" (UID: \"ddae13eb-94cf-4d57-8d16-0807a8c63423\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh"
Sep 11 16:09:56.844444 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.844285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncdjl\" (UniqueName: \"kubernetes.io/projected/31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e-kube-api-access-ncdjl\") pod \"limitador-operator-controller-manager-74495c69f8-pqfnf\" (UID: \"31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf"
Sep 11 16:09:56.844444 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.844341    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddae13eb-94cf-4d57-8d16-0807a8c63423-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lfdnh\" (UID: \"ddae13eb-94cf-4d57-8d16-0807a8c63423\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh"
Sep 11 16:09:56.911976 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.911952    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd"
Sep 11 16:09:56.914428 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:09:56.914401    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-6j489\" is forbidden: User \"system:node:ip-10-0-139-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-140.ec2.internal' and this object" podUID="c0ec6069-5ff9-4a99-9c9e-9725f11467e4" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"
Sep 11 16:09:56.914707 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.914693    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"
Sep 11 16:09:56.917192 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:09:56.917165    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-6j489\" is forbidden: User \"system:node:ip-10-0-139-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-140.ec2.internal' and this object" podUID="c0ec6069-5ff9-4a99-9c9e-9725f11467e4" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"
Sep 11 16:09:56.945706 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.945634    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmf69\" (UniqueName: \"kubernetes.io/projected/ddae13eb-94cf-4d57-8d16-0807a8c63423-kube-api-access-tmf69\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lfdnh\" (UID: \"ddae13eb-94cf-4d57-8d16-0807a8c63423\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh"
Sep 11 16:09:56.945706 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.945673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncdjl\" (UniqueName: \"kubernetes.io/projected/31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e-kube-api-access-ncdjl\") pod \"limitador-operator-controller-manager-74495c69f8-pqfnf\" (UID: \"31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf"
Sep 11 16:09:56.945706 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.945693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddae13eb-94cf-4d57-8d16-0807a8c63423-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lfdnh\" (UID: \"ddae13eb-94cf-4d57-8d16-0807a8c63423\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh"
Sep 11 16:09:56.945992 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.945977    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddae13eb-94cf-4d57-8d16-0807a8c63423-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lfdnh\" (UID: \"ddae13eb-94cf-4d57-8d16-0807a8c63423\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh"
Sep 11 16:09:56.956443 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.956415    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncdjl\" (UniqueName: \"kubernetes.io/projected/31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e-kube-api-access-ncdjl\") pod \"limitador-operator-controller-manager-74495c69f8-pqfnf\" (UID: \"31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf"
Sep 11 16:09:56.956746 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:56.956729    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmf69\" (UniqueName: \"kubernetes.io/projected/ddae13eb-94cf-4d57-8d16-0807a8c63423-kube-api-access-tmf69\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lfdnh\" (UID: \"ddae13eb-94cf-4d57-8d16-0807a8c63423\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh"
Sep 11 16:09:57.045471 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.045430    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh"
Sep 11 16:09:57.046637 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.046614    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvmc5\" (UniqueName: \"kubernetes.io/projected/c0b2ff8b-a631-4917-a6a6-a2d06654aadc-kube-api-access-hvmc5\") pod \"c0b2ff8b-a631-4917-a6a6-a2d06654aadc\" (UID: \"c0b2ff8b-a631-4917-a6a6-a2d06654aadc\") "
Sep 11 16:09:57.046732 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.046688    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0b2ff8b-a631-4917-a6a6-a2d06654aadc-extensions-socket-volume\") pod \"c0b2ff8b-a631-4917-a6a6-a2d06654aadc\" (UID: \"c0b2ff8b-a631-4917-a6a6-a2d06654aadc\") "
Sep 11 16:09:57.046774 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.046736    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngj5x\" (UniqueName: \"kubernetes.io/projected/c0ec6069-5ff9-4a99-9c9e-9725f11467e4-kube-api-access-ngj5x\") pod \"c0ec6069-5ff9-4a99-9c9e-9725f11467e4\" (UID: \"c0ec6069-5ff9-4a99-9c9e-9725f11467e4\") "
Sep 11 16:09:57.047242 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.047217    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b2ff8b-a631-4917-a6a6-a2d06654aadc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c0b2ff8b-a631-4917-a6a6-a2d06654aadc" (UID: "c0b2ff8b-a631-4917-a6a6-a2d06654aadc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:09:57.049077 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.049013    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ec6069-5ff9-4a99-9c9e-9725f11467e4-kube-api-access-ngj5x" (OuterVolumeSpecName: "kube-api-access-ngj5x") pod "c0ec6069-5ff9-4a99-9c9e-9725f11467e4" (UID: "c0ec6069-5ff9-4a99-9c9e-9725f11467e4"). InnerVolumeSpecName "kube-api-access-ngj5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:09:57.049176 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.049073    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b2ff8b-a631-4917-a6a6-a2d06654aadc-kube-api-access-hvmc5" (OuterVolumeSpecName: "kube-api-access-hvmc5") pod "c0b2ff8b-a631-4917-a6a6-a2d06654aadc" (UID: "c0b2ff8b-a631-4917-a6a6-a2d06654aadc"). InnerVolumeSpecName "kube-api-access-hvmc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:09:57.052217 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.052197    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf"
Sep 11 16:09:57.148027 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.147993    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngj5x\" (UniqueName: \"kubernetes.io/projected/c0ec6069-5ff9-4a99-9c9e-9725f11467e4-kube-api-access-ngj5x\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:09:57.148027 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.148048    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvmc5\" (UniqueName: \"kubernetes.io/projected/c0b2ff8b-a631-4917-a6a6-a2d06654aadc-kube-api-access-hvmc5\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:09:57.148220 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.148066    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0b2ff8b-a631-4917-a6a6-a2d06654aadc-extensions-socket-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:09:57.336140 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:09:57.336111    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddae13eb_94cf_4d57_8d16_0807a8c63423.slice/crio-7beacca8195718e074d7471b8428c218fd3a1686f8c6b239eb41c412e637bb63 WatchSource:0}: Error finding container 7beacca8195718e074d7471b8428c218fd3a1686f8c6b239eb41c412e637bb63: Status 404 returned error can't find the container with id 7beacca8195718e074d7471b8428c218fd3a1686f8c6b239eb41c412e637bb63
Sep 11 16:09:57.339895 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.339872    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh"]
Sep 11 16:09:57.350496 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.350467    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf"]
Sep 11 16:09:57.353757 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:09:57.353732    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e7d2d8_7ec6_4ef0_8c87_fb40b09f761e.slice/crio-67affc1e9d02cc9bca6afac58eb0f347e5a66127558d59d26289935158c15f13 WatchSource:0}: Error finding container 67affc1e9d02cc9bca6afac58eb0f347e5a66127558d59d26289935158c15f13: Status 404 returned error can't find the container with id 67affc1e9d02cc9bca6afac58eb0f347e5a66127558d59d26289935158c15f13
Sep 11 16:09:57.483493 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.483463    2577 generic.go:356] "Generic (PLEG): container finished" podID="c0b2ff8b-a631-4917-a6a6-a2d06654aadc" containerID="a5a1668a677563b224f72431adf574d84e9c0a7b762b43b12985430b102f1280" exitCode=0
Sep 11 16:09:57.483924 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.483504    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kdrfd"
Sep 11 16:09:57.483924 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.483569    2577 scope.go:117] "RemoveContainer" containerID="a5a1668a677563b224f72431adf574d84e9c0a7b762b43b12985430b102f1280"
Sep 11 16:09:57.484911 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.484889    2577 generic.go:356] "Generic (PLEG): container finished" podID="c0ec6069-5ff9-4a99-9c9e-9725f11467e4" containerID="532f8986e406525f183381a9d5033c313cfc3cce6bc0f99425d3359448cf0f52" exitCode=0
Sep 11 16:09:57.484994 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.484934    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"
Sep 11 16:09:57.487069 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.487025    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh" event={"ID":"ddae13eb-94cf-4d57-8d16-0807a8c63423","Type":"ContainerStarted","Data":"37f00b1b1e9f3089213400c8a6e808f3a4cb4954d51b75a655ffff002f2b8a76"}
Sep 11 16:09:57.487161 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.487135    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh"
Sep 11 16:09:57.487227 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.487162    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh" event={"ID":"ddae13eb-94cf-4d57-8d16-0807a8c63423","Type":"ContainerStarted","Data":"7beacca8195718e074d7471b8428c218fd3a1686f8c6b239eb41c412e637bb63"}
Sep 11 16:09:57.489022 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:09:57.488990    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-6j489\" is forbidden: User \"system:node:ip-10-0-139-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-140.ec2.internal' and this object" podUID="c0ec6069-5ff9-4a99-9c9e-9725f11467e4" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"
Sep 11 16:09:57.489248 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.489228    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf" event={"ID":"31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e","Type":"ContainerStarted","Data":"71647e42bbdde284a167e127caf5ada0c031bdf861571b2ac66782dd57051e9c"}
Sep 11 16:09:57.489331 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.489272    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf" event={"ID":"31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e","Type":"ContainerStarted","Data":"67affc1e9d02cc9bca6afac58eb0f347e5a66127558d59d26289935158c15f13"}
Sep 11 16:09:57.489331 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.489300    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf"
Sep 11 16:09:57.493456 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.493425    2577 scope.go:117] "RemoveContainer" containerID="a5a1668a677563b224f72431adf574d84e9c0a7b762b43b12985430b102f1280"
Sep 11 16:09:57.493762 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:09:57.493738    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a1668a677563b224f72431adf574d84e9c0a7b762b43b12985430b102f1280\": container with ID starting with a5a1668a677563b224f72431adf574d84e9c0a7b762b43b12985430b102f1280 not found: ID does not exist" containerID="a5a1668a677563b224f72431adf574d84e9c0a7b762b43b12985430b102f1280"
Sep 11 16:09:57.493849 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.493769    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a1668a677563b224f72431adf574d84e9c0a7b762b43b12985430b102f1280"} err="failed to get container status \"a5a1668a677563b224f72431adf574d84e9c0a7b762b43b12985430b102f1280\": rpc error: code = NotFound desc = could not find container \"a5a1668a677563b224f72431adf574d84e9c0a7b762b43b12985430b102f1280\": container with ID starting with a5a1668a677563b224f72431adf574d84e9c0a7b762b43b12985430b102f1280 not found: ID does not exist"
Sep 11 16:09:57.493849 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.493787    2577 scope.go:117] "RemoveContainer" containerID="532f8986e406525f183381a9d5033c313cfc3cce6bc0f99425d3359448cf0f52"
Sep 11 16:09:57.502156 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.502121    2577 scope.go:117] "RemoveContainer" containerID="532f8986e406525f183381a9d5033c313cfc3cce6bc0f99425d3359448cf0f52"
Sep 11 16:09:57.502439 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:09:57.502417    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"532f8986e406525f183381a9d5033c313cfc3cce6bc0f99425d3359448cf0f52\": container with ID starting with 532f8986e406525f183381a9d5033c313cfc3cce6bc0f99425d3359448cf0f52 not found: ID does not exist" containerID="532f8986e406525f183381a9d5033c313cfc3cce6bc0f99425d3359448cf0f52"
Sep 11 16:09:57.502502 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.502447    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"532f8986e406525f183381a9d5033c313cfc3cce6bc0f99425d3359448cf0f52"} err="failed to get container status \"532f8986e406525f183381a9d5033c313cfc3cce6bc0f99425d3359448cf0f52\": rpc error: code = NotFound desc = could not find container \"532f8986e406525f183381a9d5033c313cfc3cce6bc0f99425d3359448cf0f52\": container with ID starting with 532f8986e406525f183381a9d5033c313cfc3cce6bc0f99425d3359448cf0f52 not found: ID does not exist"
Sep 11 16:09:57.534722 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:09:57.534667    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-6j489\" is forbidden: User \"system:node:ip-10-0-139-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-140.ec2.internal' and this object" podUID="c0ec6069-5ff9-4a99-9c9e-9725f11467e4" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"
Sep 11 16:09:57.535587 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.535540    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh" podStartSLOduration=1.535524003 podStartE2EDuration="1.535524003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:57.531588909 +0000 UTC m=+457.377232462" watchObservedRunningTime="2026-09-11 16:09:57.535524003 +0000 UTC m=+457.381167556"
Sep 11 16:09:57.537814 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:09:57.537787    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-6j489\" is forbidden: User \"system:node:ip-10-0-139-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-140.ec2.internal' and this object" podUID="c0ec6069-5ff9-4a99-9c9e-9725f11467e4" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"
Sep 11 16:09:57.565131 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:57.565084    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf" podStartSLOduration=1.565068398 podStartE2EDuration="1.565068398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:57.563231446 +0000 UTC m=+457.408874999" watchObservedRunningTime="2026-09-11 16:09:57.565068398 +0000 UTC m=+457.410711948"
Sep 11 16:09:58.449651 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:58.449612    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm"]
Sep 11 16:09:58.454413 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:58.454396    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm"
Sep 11 16:09:58.461194 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:09:58.461165    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-6j489\" is forbidden: User \"system:node:ip-10-0-139-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-140.ec2.internal' and this object" podUID="c0ec6069-5ff9-4a99-9c9e-9725f11467e4" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6j489"
Sep 11 16:09:58.479143 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:58.479113    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm"]
Sep 11 16:09:58.558844 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:58.558806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8cvd\" (UniqueName: \"kubernetes.io/projected/96a83de2-03be-4483-a321-7f71b53a3a1b-kube-api-access-w8cvd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6npsm\" (UID: \"96a83de2-03be-4483-a321-7f71b53a3a1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm"
Sep 11 16:09:58.558844 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:58.558848    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/96a83de2-03be-4483-a321-7f71b53a3a1b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6npsm\" (UID: \"96a83de2-03be-4483-a321-7f71b53a3a1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm"
Sep 11 16:09:58.631661 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:58.631631    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0b2ff8b-a631-4917-a6a6-a2d06654aadc" path="/var/lib/kubelet/pods/c0b2ff8b-a631-4917-a6a6-a2d06654aadc/volumes"
Sep 11 16:09:58.631953 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:58.631941    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0ec6069-5ff9-4a99-9c9e-9725f11467e4" path="/var/lib/kubelet/pods/c0ec6069-5ff9-4a99-9c9e-9725f11467e4/volumes"
Sep 11 16:09:58.660195 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:58.660160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/96a83de2-03be-4483-a321-7f71b53a3a1b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6npsm\" (UID: \"96a83de2-03be-4483-a321-7f71b53a3a1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm"
Sep 11 16:09:58.660309 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:58.660278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8cvd\" (UniqueName: \"kubernetes.io/projected/96a83de2-03be-4483-a321-7f71b53a3a1b-kube-api-access-w8cvd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6npsm\" (UID: \"96a83de2-03be-4483-a321-7f71b53a3a1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm"
Sep 11 16:09:58.660688 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:58.660668    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/96a83de2-03be-4483-a321-7f71b53a3a1b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6npsm\" (UID: \"96a83de2-03be-4483-a321-7f71b53a3a1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm"
Sep 11 16:09:58.675695 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:58.675670    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8cvd\" (UniqueName: \"kubernetes.io/projected/96a83de2-03be-4483-a321-7f71b53a3a1b-kube-api-access-w8cvd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6npsm\" (UID: \"96a83de2-03be-4483-a321-7f71b53a3a1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm"
Sep 11 16:09:58.764085 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:58.763987    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm"
Sep 11 16:09:59.071916 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:59.071831    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm"]
Sep 11 16:09:59.076205 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:09:59.076168    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a83de2_03be_4483_a321_7f71b53a3a1b.slice/crio-9f4e0814dd27c9e926ddcd61e005663ad49e4c1c3eb1b8cf4041951e0c43ffce WatchSource:0}: Error finding container 9f4e0814dd27c9e926ddcd61e005663ad49e4c1c3eb1b8cf4041951e0c43ffce: Status 404 returned error can't find the container with id 9f4e0814dd27c9e926ddcd61e005663ad49e4c1c3eb1b8cf4041951e0c43ffce
Sep 11 16:09:59.504502 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:59.504466    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm" event={"ID":"96a83de2-03be-4483-a321-7f71b53a3a1b","Type":"ContainerStarted","Data":"8ed23ec73f42eefb2cd9d28e11bf9dc4f8a5f6a635664b267b7e117a8be0032a"}
Sep 11 16:09:59.504653 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:59.504549    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm" event={"ID":"96a83de2-03be-4483-a321-7f71b53a3a1b","Type":"ContainerStarted","Data":"9f4e0814dd27c9e926ddcd61e005663ad49e4c1c3eb1b8cf4041951e0c43ffce"}
Sep 11 16:09:59.504653 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:59.504617    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm"
Sep 11 16:09:59.533634 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:09:59.533579    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm" podStartSLOduration=1.5335502989999998 podStartE2EDuration="1.533550299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:59.531219075 +0000 UTC m=+459.376862651" watchObservedRunningTime="2026-09-11 16:09:59.533550299 +0000 UTC m=+459.379193853"
Sep 11 16:10:08.499799 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:08.499761    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh"
Sep 11 16:10:08.500177 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:08.499837    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf"
Sep 11 16:10:10.511886 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:10.511856    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm"
Sep 11 16:10:10.580674 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:10.580641    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh"]
Sep 11 16:10:10.580837 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:10.580817    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh" podUID="ddae13eb-94cf-4d57-8d16-0807a8c63423" containerName="manager" containerID="cri-o://37f00b1b1e9f3089213400c8a6e808f3a4cb4954d51b75a655ffff002f2b8a76" gracePeriod=10
Sep 11 16:10:10.831803 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:10.831777    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh"
Sep 11 16:10:10.958569 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:10.958534    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmf69\" (UniqueName: \"kubernetes.io/projected/ddae13eb-94cf-4d57-8d16-0807a8c63423-kube-api-access-tmf69\") pod \"ddae13eb-94cf-4d57-8d16-0807a8c63423\" (UID: \"ddae13eb-94cf-4d57-8d16-0807a8c63423\") "
Sep 11 16:10:10.958764 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:10.958607    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddae13eb-94cf-4d57-8d16-0807a8c63423-extensions-socket-volume\") pod \"ddae13eb-94cf-4d57-8d16-0807a8c63423\" (UID: \"ddae13eb-94cf-4d57-8d16-0807a8c63423\") "
Sep 11 16:10:10.959098 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:10.959066    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddae13eb-94cf-4d57-8d16-0807a8c63423-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ddae13eb-94cf-4d57-8d16-0807a8c63423" (UID: "ddae13eb-94cf-4d57-8d16-0807a8c63423"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:10:10.961060 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:10.961011    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddae13eb-94cf-4d57-8d16-0807a8c63423-kube-api-access-tmf69" (OuterVolumeSpecName: "kube-api-access-tmf69") pod "ddae13eb-94cf-4d57-8d16-0807a8c63423" (UID: "ddae13eb-94cf-4d57-8d16-0807a8c63423"). InnerVolumeSpecName "kube-api-access-tmf69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:10:11.059754 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:11.059716    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmf69\" (UniqueName: \"kubernetes.io/projected/ddae13eb-94cf-4d57-8d16-0807a8c63423-kube-api-access-tmf69\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:10:11.059754 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:11.059745    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddae13eb-94cf-4d57-8d16-0807a8c63423-extensions-socket-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:10:11.563199 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:11.563164    2577 generic.go:356] "Generic (PLEG): container finished" podID="ddae13eb-94cf-4d57-8d16-0807a8c63423" containerID="37f00b1b1e9f3089213400c8a6e808f3a4cb4954d51b75a655ffff002f2b8a76" exitCode=0
Sep 11 16:10:11.563674 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:11.563210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh" event={"ID":"ddae13eb-94cf-4d57-8d16-0807a8c63423","Type":"ContainerDied","Data":"37f00b1b1e9f3089213400c8a6e808f3a4cb4954d51b75a655ffff002f2b8a76"}
Sep 11 16:10:11.563674 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:11.563239    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh"
Sep 11 16:10:11.563674 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:11.563288    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh" event={"ID":"ddae13eb-94cf-4d57-8d16-0807a8c63423","Type":"ContainerDied","Data":"7beacca8195718e074d7471b8428c218fd3a1686f8c6b239eb41c412e637bb63"}
Sep 11 16:10:11.563674 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:11.563308    2577 scope.go:117] "RemoveContainer" containerID="37f00b1b1e9f3089213400c8a6e808f3a4cb4954d51b75a655ffff002f2b8a76"
Sep 11 16:10:11.571687 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:11.571662    2577 scope.go:117] "RemoveContainer" containerID="37f00b1b1e9f3089213400c8a6e808f3a4cb4954d51b75a655ffff002f2b8a76"
Sep 11 16:10:11.571968 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:10:11.571949    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f00b1b1e9f3089213400c8a6e808f3a4cb4954d51b75a655ffff002f2b8a76\": container with ID starting with 37f00b1b1e9f3089213400c8a6e808f3a4cb4954d51b75a655ffff002f2b8a76 not found: ID does not exist" containerID="37f00b1b1e9f3089213400c8a6e808f3a4cb4954d51b75a655ffff002f2b8a76"
Sep 11 16:10:11.572006 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:11.571977    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f00b1b1e9f3089213400c8a6e808f3a4cb4954d51b75a655ffff002f2b8a76"} err="failed to get container status \"37f00b1b1e9f3089213400c8a6e808f3a4cb4954d51b75a655ffff002f2b8a76\": rpc error: code = NotFound desc = could not find container \"37f00b1b1e9f3089213400c8a6e808f3a4cb4954d51b75a655ffff002f2b8a76\": container with ID starting with 37f00b1b1e9f3089213400c8a6e808f3a4cb4954d51b75a655ffff002f2b8a76 not found: ID does not exist"
Sep 11 16:10:11.585647 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:11.585620    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh"]
Sep 11 16:10:11.591851 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:11.591827    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lfdnh"]
Sep 11 16:10:12.632482 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:12.632451    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddae13eb-94cf-4d57-8d16-0807a8c63423" path="/var/lib/kubelet/pods/ddae13eb-94cf-4d57-8d16-0807a8c63423/volumes"
Sep 11 16:10:29.265498 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:29.265460    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-llk9r"]
Sep 11 16:10:29.265984 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:29.265966    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddae13eb-94cf-4d57-8d16-0807a8c63423" containerName="manager"
Sep 11 16:10:29.266067 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:29.265984    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddae13eb-94cf-4d57-8d16-0807a8c63423" containerName="manager"
Sep 11 16:10:29.266137 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:29.266112    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddae13eb-94cf-4d57-8d16-0807a8c63423" containerName="manager"
Sep 11 16:10:29.269028 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:29.269002    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-llk9r"
Sep 11 16:10:29.287872 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:29.287849    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 16:10:29.288659 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:29.288638    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-b99gd\""
Sep 11 16:10:29.299297 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:29.299234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjzkt\" (UniqueName: \"kubernetes.io/projected/c4b6a725-dc18-468d-b1d7-41489b79c4af-kube-api-access-fjzkt\") pod \"limitador-limitador-8588749977-llk9r\" (UID: \"c4b6a725-dc18-468d-b1d7-41489b79c4af\") " pod="kuadrant-system/limitador-limitador-8588749977-llk9r"
Sep 11 16:10:29.299463 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:29.299306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4b6a725-dc18-468d-b1d7-41489b79c4af-config-file\") pod \"limitador-limitador-8588749977-llk9r\" (UID: \"c4b6a725-dc18-468d-b1d7-41489b79c4af\") " pod="kuadrant-system/limitador-limitador-8588749977-llk9r"
Sep 11 16:10:29.302385 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:29.302352    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-llk9r"]
Sep 11 16:10:29.387528 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:29.387455    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-llk9r"]
Sep 11 16:10:29.399915 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:29.399886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjzkt\" (UniqueName: \"kubernetes.io/projected/c4b6a725-dc18-468d-b1d7-41489b79c4af-kube-api-access-fjzkt\") pod \"limitador-limitador-8588749977-llk9r\" (UID: \"c4b6a725-dc18-468d-b1d7-41489b79c4af\") " pod="kuadrant-system/limitador-limitador-8588749977-llk9r"
Sep 11 16:10:29.400082 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:29.399946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4b6a725-dc18-468d-b1d7-41489b79c4af-config-file\") pod \"limitador-limitador-8588749977-llk9r\" (UID: \"c4b6a725-dc18-468d-b1d7-41489b79c4af\") " pod="kuadrant-system/limitador-limitador-8588749977-llk9r"
Sep 11 16:10:29.400653 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:29.400621    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4b6a725-dc18-468d-b1d7-41489b79c4af-config-file\") pod \"limitador-limitador-8588749977-llk9r\" (UID: \"c4b6a725-dc18-468d-b1d7-41489b79c4af\") " pod="kuadrant-system/limitador-limitador-8588749977-llk9r"
Sep 11 16:10:29.416001 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:29.415967    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjzkt\" (UniqueName: \"kubernetes.io/projected/c4b6a725-dc18-468d-b1d7-41489b79c4af-kube-api-access-fjzkt\") pod \"limitador-limitador-8588749977-llk9r\" (UID: \"c4b6a725-dc18-468d-b1d7-41489b79c4af\") " pod="kuadrant-system/limitador-limitador-8588749977-llk9r"
Sep 11 16:10:29.581459 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:29.581366    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-llk9r"
Sep 11 16:10:29.866790 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:29.866768    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-llk9r"]
Sep 11 16:10:29.868741 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:10:29.868715    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b6a725_dc18_468d_b1d7_41489b79c4af.slice/crio-0549819b8b4ab4266aa01b495c57f2a5519c98a2ef91082dab40ce316318e2f6 WatchSource:0}: Error finding container 0549819b8b4ab4266aa01b495c57f2a5519c98a2ef91082dab40ce316318e2f6: Status 404 returned error can't find the container with id 0549819b8b4ab4266aa01b495c57f2a5519c98a2ef91082dab40ce316318e2f6
Sep 11 16:10:30.317015 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:30.316928    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-s6b4l"]
Sep 11 16:10:30.319551 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:30.319521    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-s6b4l"
Sep 11 16:10:30.323315 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:30.323203    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tvg9c\""
Sep 11 16:10:30.339990 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:30.339956    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-s6b4l"]
Sep 11 16:10:30.407325 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:30.407290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg25v\" (UniqueName: \"kubernetes.io/projected/1300844c-e568-4be9-aaaa-c807f75a8c97-kube-api-access-xg25v\") pod \"authorino-7bcfd8f9cd-s6b4l\" (UID: \"1300844c-e568-4be9-aaaa-c807f75a8c97\") " pod="kuadrant-system/authorino-7bcfd8f9cd-s6b4l"
Sep 11 16:10:30.508673 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:30.508638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg25v\" (UniqueName: \"kubernetes.io/projected/1300844c-e568-4be9-aaaa-c807f75a8c97-kube-api-access-xg25v\") pod \"authorino-7bcfd8f9cd-s6b4l\" (UID: \"1300844c-e568-4be9-aaaa-c807f75a8c97\") " pod="kuadrant-system/authorino-7bcfd8f9cd-s6b4l"
Sep 11 16:10:30.522017 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:30.521987    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg25v\" (UniqueName: \"kubernetes.io/projected/1300844c-e568-4be9-aaaa-c807f75a8c97-kube-api-access-xg25v\") pod \"authorino-7bcfd8f9cd-s6b4l\" (UID: \"1300844c-e568-4be9-aaaa-c807f75a8c97\") " pod="kuadrant-system/authorino-7bcfd8f9cd-s6b4l"
Sep 11 16:10:30.629063 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:30.629017    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-s6b4l"
Sep 11 16:10:30.655565 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:30.655528    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-llk9r" event={"ID":"c4b6a725-dc18-468d-b1d7-41489b79c4af","Type":"ContainerStarted","Data":"0549819b8b4ab4266aa01b495c57f2a5519c98a2ef91082dab40ce316318e2f6"}
Sep 11 16:10:30.751928 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:30.751895    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-kqmgp"]
Sep 11 16:10:30.753926 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:30.753910    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-kqmgp"
Sep 11 16:10:30.769179 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:30.769152    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-kqmgp"]
Sep 11 16:10:30.811426 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:30.811386    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgztj\" (UniqueName: \"kubernetes.io/projected/f393a664-8c41-41d8-98d5-cd1d120f8f31-kube-api-access-jgztj\") pod \"authorino-59c974bd48-kqmgp\" (UID: \"f393a664-8c41-41d8-98d5-cd1d120f8f31\") " pod="kuadrant-system/authorino-59c974bd48-kqmgp"
Sep 11 16:10:30.912151 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:30.912029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgztj\" (UniqueName: \"kubernetes.io/projected/f393a664-8c41-41d8-98d5-cd1d120f8f31-kube-api-access-jgztj\") pod \"authorino-59c974bd48-kqmgp\" (UID: \"f393a664-8c41-41d8-98d5-cd1d120f8f31\") " pod="kuadrant-system/authorino-59c974bd48-kqmgp"
Sep 11 16:10:30.924727 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:30.924689    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgztj\" (UniqueName: \"kubernetes.io/projected/f393a664-8c41-41d8-98d5-cd1d120f8f31-kube-api-access-jgztj\") pod \"authorino-59c974bd48-kqmgp\" (UID: \"f393a664-8c41-41d8-98d5-cd1d120f8f31\") " pod="kuadrant-system/authorino-59c974bd48-kqmgp"
Sep 11 16:10:30.928792 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:30.928767    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-s6b4l"]
Sep 11 16:10:31.063428 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:31.063386    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-kqmgp"
Sep 11 16:10:31.085625 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:10:31.085594    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1300844c_e568_4be9_aaaa_c807f75a8c97.slice/crio-d2606eb5dffd5dbdc8e4a433c793bd56ab0bb7d8d6e4c78350709f55519cfdbf WatchSource:0}: Error finding container d2606eb5dffd5dbdc8e4a433c793bd56ab0bb7d8d6e4c78350709f55519cfdbf: Status 404 returned error can't find the container with id d2606eb5dffd5dbdc8e4a433c793bd56ab0bb7d8d6e4c78350709f55519cfdbf
Sep 11 16:10:31.366772 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:31.366736    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-kqmgp"]
Sep 11 16:10:31.370199 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:10:31.370172    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf393a664_8c41_41d8_98d5_cd1d120f8f31.slice/crio-d6364a4afe33ae058f833321c7d53dc7b9b7e1724be210a040d5f9ff492683e5 WatchSource:0}: Error finding container d6364a4afe33ae058f833321c7d53dc7b9b7e1724be210a040d5f9ff492683e5: Status 404 returned error can't find the container with id d6364a4afe33ae058f833321c7d53dc7b9b7e1724be210a040d5f9ff492683e5
Sep 11 16:10:31.660616 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:31.660579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-s6b4l" event={"ID":"1300844c-e568-4be9-aaaa-c807f75a8c97","Type":"ContainerStarted","Data":"d2606eb5dffd5dbdc8e4a433c793bd56ab0bb7d8d6e4c78350709f55519cfdbf"}
Sep 11 16:10:31.661932 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:31.661906    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-kqmgp" event={"ID":"f393a664-8c41-41d8-98d5-cd1d120f8f31","Type":"ContainerStarted","Data":"d6364a4afe33ae058f833321c7d53dc7b9b7e1724be210a040d5f9ff492683e5"}
Sep 11 16:10:31.663399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:31.663380    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-llk9r" event={"ID":"c4b6a725-dc18-468d-b1d7-41489b79c4af","Type":"ContainerStarted","Data":"9a6c9840a145e17e654b8985a762f9741726c5f94b351786e55b352cf8641822"}
Sep 11 16:10:31.663533 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:31.663522    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-llk9r"
Sep 11 16:10:31.713337 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:31.713274    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-llk9r" podStartSLOduration=1.470985736 podStartE2EDuration="2.71325212s" totalImagesPullingTime="1.242266384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:10:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:10:31.711933437 +0000 UTC m=+491.557576992" watchObservedRunningTime="2026-09-11 16:10:31.71325212 +0000 UTC m=+491.558895675"
Sep 11 16:10:33.675517 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:33.675479    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-s6b4l" event={"ID":"1300844c-e568-4be9-aaaa-c807f75a8c97","Type":"ContainerStarted","Data":"ddd94f2fac5a368927e7cd785ac952fdd02d8f743290d94c09c476c1bc3f9496"}
Sep 11 16:10:33.677095 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:33.677070    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-kqmgp" event={"ID":"f393a664-8c41-41d8-98d5-cd1d120f8f31","Type":"ContainerStarted","Data":"42eef37ac91610fb0a5f6113c1132c8959fc54299b462307691fdb596c6d1cd4"}
Sep 11 16:10:33.694784 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:33.694725    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-s6b4l" podStartSLOduration=2.057965274 podStartE2EDuration="3.694704144s" totalImagesPullingTime="1.63673887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:10:33.693196122 +0000 UTC m=+493.538839679" watchObservedRunningTime="2026-09-11 16:10:33.694704144 +0000 UTC m=+493.540347699"
Sep 11 16:10:33.712205 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:33.712158    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-kqmgp" podStartSLOduration=2.355806955 podStartE2EDuration="3.712139154s" totalImagesPullingTime="1.356332199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:10:33.71079608 +0000 UTC m=+493.556439636" watchObservedRunningTime="2026-09-11 16:10:33.712139154 +0000 UTC m=+493.557782709"
Sep 11 16:10:33.742703 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:33.742659    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-s6b4l"]
Sep 11 16:10:35.687475 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:35.687408    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-s6b4l" podUID="1300844c-e568-4be9-aaaa-c807f75a8c97" containerName="authorino" containerID="cri-o://ddd94f2fac5a368927e7cd785ac952fdd02d8f743290d94c09c476c1bc3f9496" gracePeriod=30
Sep 11 16:10:35.927502 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:35.927472    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-s6b4l"
Sep 11 16:10:36.056853 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:36.056760    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg25v\" (UniqueName: \"kubernetes.io/projected/1300844c-e568-4be9-aaaa-c807f75a8c97-kube-api-access-xg25v\") pod \"1300844c-e568-4be9-aaaa-c807f75a8c97\" (UID: \"1300844c-e568-4be9-aaaa-c807f75a8c97\") "
Sep 11 16:10:36.059090 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:36.059062    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1300844c-e568-4be9-aaaa-c807f75a8c97-kube-api-access-xg25v" (OuterVolumeSpecName: "kube-api-access-xg25v") pod "1300844c-e568-4be9-aaaa-c807f75a8c97" (UID: "1300844c-e568-4be9-aaaa-c807f75a8c97"). InnerVolumeSpecName "kube-api-access-xg25v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:10:36.158256 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:36.158217    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg25v\" (UniqueName: \"kubernetes.io/projected/1300844c-e568-4be9-aaaa-c807f75a8c97-kube-api-access-xg25v\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:10:36.692462 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:36.692427    2577 generic.go:356] "Generic (PLEG): container finished" podID="1300844c-e568-4be9-aaaa-c807f75a8c97" containerID="ddd94f2fac5a368927e7cd785ac952fdd02d8f743290d94c09c476c1bc3f9496" exitCode=0
Sep 11 16:10:36.692889 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:36.692471    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-s6b4l"
Sep 11 16:10:36.692889 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:36.692517    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-s6b4l" event={"ID":"1300844c-e568-4be9-aaaa-c807f75a8c97","Type":"ContainerDied","Data":"ddd94f2fac5a368927e7cd785ac952fdd02d8f743290d94c09c476c1bc3f9496"}
Sep 11 16:10:36.692889 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:36.692555    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-s6b4l" event={"ID":"1300844c-e568-4be9-aaaa-c807f75a8c97","Type":"ContainerDied","Data":"d2606eb5dffd5dbdc8e4a433c793bd56ab0bb7d8d6e4c78350709f55519cfdbf"}
Sep 11 16:10:36.692889 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:36.692571    2577 scope.go:117] "RemoveContainer" containerID="ddd94f2fac5a368927e7cd785ac952fdd02d8f743290d94c09c476c1bc3f9496"
Sep 11 16:10:36.701224 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:36.701206    2577 scope.go:117] "RemoveContainer" containerID="ddd94f2fac5a368927e7cd785ac952fdd02d8f743290d94c09c476c1bc3f9496"
Sep 11 16:10:36.701507 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:10:36.701490    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddd94f2fac5a368927e7cd785ac952fdd02d8f743290d94c09c476c1bc3f9496\": container with ID starting with ddd94f2fac5a368927e7cd785ac952fdd02d8f743290d94c09c476c1bc3f9496 not found: ID does not exist" containerID="ddd94f2fac5a368927e7cd785ac952fdd02d8f743290d94c09c476c1bc3f9496"
Sep 11 16:10:36.701556 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:36.701519    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddd94f2fac5a368927e7cd785ac952fdd02d8f743290d94c09c476c1bc3f9496"} err="failed to get container status \"ddd94f2fac5a368927e7cd785ac952fdd02d8f743290d94c09c476c1bc3f9496\": rpc error: code = NotFound desc = could not find container \"ddd94f2fac5a368927e7cd785ac952fdd02d8f743290d94c09c476c1bc3f9496\": container with ID starting with ddd94f2fac5a368927e7cd785ac952fdd02d8f743290d94c09c476c1bc3f9496 not found: ID does not exist"
Sep 11 16:10:36.709930 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:36.709899    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-s6b4l"]
Sep 11 16:10:36.714341 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:36.714318    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-s6b4l"]
Sep 11 16:10:38.632453 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:38.632422    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1300844c-e568-4be9-aaaa-c807f75a8c97" path="/var/lib/kubelet/pods/1300844c-e568-4be9-aaaa-c807f75a8c97/volumes"
Sep 11 16:10:42.670639 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:42.670612    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-llk9r"
Sep 11 16:10:43.827598 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:43.827567    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-llk9r"]
Sep 11 16:10:43.827993 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:43.827747    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-llk9r" podUID="c4b6a725-dc18-468d-b1d7-41489b79c4af" containerName="limitador" containerID="cri-o://9a6c9840a145e17e654b8985a762f9741726c5f94b351786e55b352cf8641822" gracePeriod=30
Sep 11 16:10:44.377648 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:44.377621    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-llk9r"
Sep 11 16:10:44.526352 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:44.526265    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjzkt\" (UniqueName: \"kubernetes.io/projected/c4b6a725-dc18-468d-b1d7-41489b79c4af-kube-api-access-fjzkt\") pod \"c4b6a725-dc18-468d-b1d7-41489b79c4af\" (UID: \"c4b6a725-dc18-468d-b1d7-41489b79c4af\") "
Sep 11 16:10:44.526352 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:44.526305    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4b6a725-dc18-468d-b1d7-41489b79c4af-config-file\") pod \"c4b6a725-dc18-468d-b1d7-41489b79c4af\" (UID: \"c4b6a725-dc18-468d-b1d7-41489b79c4af\") "
Sep 11 16:10:44.526742 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:44.526713    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b6a725-dc18-468d-b1d7-41489b79c4af-config-file" (OuterVolumeSpecName: "config-file") pod "c4b6a725-dc18-468d-b1d7-41489b79c4af" (UID: "c4b6a725-dc18-468d-b1d7-41489b79c4af"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:10:44.528607 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:44.528581    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b6a725-dc18-468d-b1d7-41489b79c4af-kube-api-access-fjzkt" (OuterVolumeSpecName: "kube-api-access-fjzkt") pod "c4b6a725-dc18-468d-b1d7-41489b79c4af" (UID: "c4b6a725-dc18-468d-b1d7-41489b79c4af"). InnerVolumeSpecName "kube-api-access-fjzkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:10:44.627464 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:44.627424    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjzkt\" (UniqueName: \"kubernetes.io/projected/c4b6a725-dc18-468d-b1d7-41489b79c4af-kube-api-access-fjzkt\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:10:44.627464 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:44.627461    2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4b6a725-dc18-468d-b1d7-41489b79c4af-config-file\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:10:44.728875 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:44.728840    2577 generic.go:356] "Generic (PLEG): container finished" podID="c4b6a725-dc18-468d-b1d7-41489b79c4af" containerID="9a6c9840a145e17e654b8985a762f9741726c5f94b351786e55b352cf8641822" exitCode=0
Sep 11 16:10:44.729075 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:44.728888    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-llk9r"
Sep 11 16:10:44.729075 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:44.728897    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-llk9r" event={"ID":"c4b6a725-dc18-468d-b1d7-41489b79c4af","Type":"ContainerDied","Data":"9a6c9840a145e17e654b8985a762f9741726c5f94b351786e55b352cf8641822"}
Sep 11 16:10:44.729075 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:44.728948    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-llk9r" event={"ID":"c4b6a725-dc18-468d-b1d7-41489b79c4af","Type":"ContainerDied","Data":"0549819b8b4ab4266aa01b495c57f2a5519c98a2ef91082dab40ce316318e2f6"}
Sep 11 16:10:44.729075 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:44.728978    2577 scope.go:117] "RemoveContainer" containerID="9a6c9840a145e17e654b8985a762f9741726c5f94b351786e55b352cf8641822"
Sep 11 16:10:44.737476 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:44.737454    2577 scope.go:117] "RemoveContainer" containerID="9a6c9840a145e17e654b8985a762f9741726c5f94b351786e55b352cf8641822"
Sep 11 16:10:44.737745 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:10:44.737726    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a6c9840a145e17e654b8985a762f9741726c5f94b351786e55b352cf8641822\": container with ID starting with 9a6c9840a145e17e654b8985a762f9741726c5f94b351786e55b352cf8641822 not found: ID does not exist" containerID="9a6c9840a145e17e654b8985a762f9741726c5f94b351786e55b352cf8641822"
Sep 11 16:10:44.737807 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:44.737752    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a6c9840a145e17e654b8985a762f9741726c5f94b351786e55b352cf8641822"} err="failed to get container status \"9a6c9840a145e17e654b8985a762f9741726c5f94b351786e55b352cf8641822\": rpc error: code = NotFound desc = could not find container \"9a6c9840a145e17e654b8985a762f9741726c5f94b351786e55b352cf8641822\": container with ID starting with 9a6c9840a145e17e654b8985a762f9741726c5f94b351786e55b352cf8641822 not found: ID does not exist"
Sep 11 16:10:44.745331 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:44.745303    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-llk9r"]
Sep 11 16:10:44.749356 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:44.749335    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-llk9r"]
Sep 11 16:10:45.704844 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:45.704803    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ll8j7"]
Sep 11 16:10:45.705223 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:45.705151    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1300844c-e568-4be9-aaaa-c807f75a8c97" containerName="authorino"
Sep 11 16:10:45.705223 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:45.705163    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1300844c-e568-4be9-aaaa-c807f75a8c97" containerName="authorino"
Sep 11 16:10:45.705223 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:45.705180    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4b6a725-dc18-468d-b1d7-41489b79c4af" containerName="limitador"
Sep 11 16:10:45.705223 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:45.705185    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4b6a725-dc18-468d-b1d7-41489b79c4af" containerName="limitador"
Sep 11 16:10:45.705354 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:45.705242    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4b6a725-dc18-468d-b1d7-41489b79c4af" containerName="limitador"
Sep 11 16:10:45.705354 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:45.705249    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1300844c-e568-4be9-aaaa-c807f75a8c97" containerName="authorino"
Sep 11 16:10:45.709313 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:45.709296    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ll8j7"
Sep 11 16:10:45.712004 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:45.711984    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 16:10:45.712276 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:45.712257    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-5lls8\""
Sep 11 16:10:45.713079 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:45.713063    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 16:10:45.713135 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:45.713097    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 11 16:10:45.724392 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:45.724360    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ll8j7"]
Sep 11 16:10:45.837089 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:45.837030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpc2x\" (UniqueName: \"kubernetes.io/projected/cdfd0296-4092-4b0c-b2db-6c748bcd32ad-kube-api-access-bpc2x\") pod \"postgres-69fb4594cf-ll8j7\" (UID: \"cdfd0296-4092-4b0c-b2db-6c748bcd32ad\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ll8j7"
Sep 11 16:10:45.837275 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:45.837115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cdfd0296-4092-4b0c-b2db-6c748bcd32ad-data\") pod \"postgres-69fb4594cf-ll8j7\" (UID: \"cdfd0296-4092-4b0c-b2db-6c748bcd32ad\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ll8j7"
Sep 11 16:10:45.938025 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:45.937987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cdfd0296-4092-4b0c-b2db-6c748bcd32ad-data\") pod \"postgres-69fb4594cf-ll8j7\" (UID: \"cdfd0296-4092-4b0c-b2db-6c748bcd32ad\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ll8j7"
Sep 11 16:10:45.938242 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:45.938074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpc2x\" (UniqueName: \"kubernetes.io/projected/cdfd0296-4092-4b0c-b2db-6c748bcd32ad-kube-api-access-bpc2x\") pod \"postgres-69fb4594cf-ll8j7\" (UID: \"cdfd0296-4092-4b0c-b2db-6c748bcd32ad\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ll8j7"
Sep 11 16:10:45.938382 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:45.938363    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cdfd0296-4092-4b0c-b2db-6c748bcd32ad-data\") pod \"postgres-69fb4594cf-ll8j7\" (UID: \"cdfd0296-4092-4b0c-b2db-6c748bcd32ad\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ll8j7"
Sep 11 16:10:45.949091 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:45.949070    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpc2x\" (UniqueName: \"kubernetes.io/projected/cdfd0296-4092-4b0c-b2db-6c748bcd32ad-kube-api-access-bpc2x\") pod \"postgres-69fb4594cf-ll8j7\" (UID: \"cdfd0296-4092-4b0c-b2db-6c748bcd32ad\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ll8j7"
Sep 11 16:10:46.019111 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:46.019003    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ll8j7"
Sep 11 16:10:46.327898 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:46.327812    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ll8j7"]
Sep 11 16:10:46.330622 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:10:46.330589    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdfd0296_4092_4b0c_b2db_6c748bcd32ad.slice/crio-b03ab14a0b6384e1802b8210a837a85af74495cb8f154d57c0dc6958ce18d6ab WatchSource:0}: Error finding container b03ab14a0b6384e1802b8210a837a85af74495cb8f154d57c0dc6958ce18d6ab: Status 404 returned error can't find the container with id b03ab14a0b6384e1802b8210a837a85af74495cb8f154d57c0dc6958ce18d6ab
Sep 11 16:10:46.631691 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:46.631659    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4b6a725-dc18-468d-b1d7-41489b79c4af" path="/var/lib/kubelet/pods/c4b6a725-dc18-468d-b1d7-41489b79c4af/volumes"
Sep 11 16:10:46.740962 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:46.740929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ll8j7" event={"ID":"cdfd0296-4092-4b0c-b2db-6c748bcd32ad","Type":"ContainerStarted","Data":"b03ab14a0b6384e1802b8210a837a85af74495cb8f154d57c0dc6958ce18d6ab"}
Sep 11 16:10:51.768813 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:51.768723    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ll8j7" event={"ID":"cdfd0296-4092-4b0c-b2db-6c748bcd32ad","Type":"ContainerStarted","Data":"00458d0ee2f0df6da7bed40507af7846b3070b21bce422f1490fc5175edb6864"}
Sep 11 16:10:51.769204 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:51.768859    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ll8j7"
Sep 11 16:10:51.789498 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:51.789412    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ll8j7" podStartSLOduration=1.7044924689999998 podStartE2EDuration="6.789381691s" totalImagesPullingTime="5.084889222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:10:51.787261635 +0000 UTC m=+511.632905191" watchObservedRunningTime="2026-09-11 16:10:51.789381691 +0000 UTC m=+511.635025247"
Sep 11 16:10:57.802683 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:57.802656    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ll8j7"
Sep 11 16:10:58.438337 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.438304    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-vrxgr"]
Sep 11 16:10:58.444213 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.444189    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-vrxgr"
Sep 11 16:10:58.461420 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.461394    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-vrxgr"]
Sep 11 16:10:58.545445 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.545406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7llb5\" (UniqueName: \"kubernetes.io/projected/14bb3596-4768-4a41-aa88-6bd2c58ec8fa-kube-api-access-7llb5\") pod \"authorino-557f876f69-vrxgr\" (UID: \"14bb3596-4768-4a41-aa88-6bd2c58ec8fa\") " pod="kuadrant-system/authorino-557f876f69-vrxgr"
Sep 11 16:10:58.549988 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.549957    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-vrxgr"]
Sep 11 16:10:58.550241 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:10:58.550221    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-7llb5], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-vrxgr" podUID="14bb3596-4768-4a41-aa88-6bd2c58ec8fa"
Sep 11 16:10:58.614159 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.614122    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-sswmp"]
Sep 11 16:10:58.618219 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.618197    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-sswmp"
Sep 11 16:10:58.621998 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.621979    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 16:10:58.633642 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.633614    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-sswmp"]
Sep 11 16:10:58.646337 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.646309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7llb5\" (UniqueName: \"kubernetes.io/projected/14bb3596-4768-4a41-aa88-6bd2c58ec8fa-kube-api-access-7llb5\") pod \"authorino-557f876f69-vrxgr\" (UID: \"14bb3596-4768-4a41-aa88-6bd2c58ec8fa\") " pod="kuadrant-system/authorino-557f876f69-vrxgr"
Sep 11 16:10:58.680591 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.680553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7llb5\" (UniqueName: \"kubernetes.io/projected/14bb3596-4768-4a41-aa88-6bd2c58ec8fa-kube-api-access-7llb5\") pod \"authorino-557f876f69-vrxgr\" (UID: \"14bb3596-4768-4a41-aa88-6bd2c58ec8fa\") " pod="kuadrant-system/authorino-557f876f69-vrxgr"
Sep 11 16:10:58.747315 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.747210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f163a5d6-43a7-43fa-b8bd-5851d4679a19-tls-cert\") pod \"authorino-6d4dfc649f-sswmp\" (UID: \"f163a5d6-43a7-43fa-b8bd-5851d4679a19\") " pod="kuadrant-system/authorino-6d4dfc649f-sswmp"
Sep 11 16:10:58.747315 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.747292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmz72\" (UniqueName: \"kubernetes.io/projected/f163a5d6-43a7-43fa-b8bd-5851d4679a19-kube-api-access-gmz72\") pod \"authorino-6d4dfc649f-sswmp\" (UID: \"f163a5d6-43a7-43fa-b8bd-5851d4679a19\") " pod="kuadrant-system/authorino-6d4dfc649f-sswmp"
Sep 11 16:10:58.764344 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.764303    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-sswmp"]
Sep 11 16:10:58.764603 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:10:58.764566    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-gmz72 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-sswmp" podUID="f163a5d6-43a7-43fa-b8bd-5851d4679a19"
Sep 11 16:10:58.800915 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.800883    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-sswmp"
Sep 11 16:10:58.801105 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.800887    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-vrxgr"
Sep 11 16:10:58.808851 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.808828    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-sswmp"
Sep 11 16:10:58.811653 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.811632    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-vrxgr"
Sep 11 16:10:58.848342 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.848309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f163a5d6-43a7-43fa-b8bd-5851d4679a19-tls-cert\") pod \"authorino-6d4dfc649f-sswmp\" (UID: \"f163a5d6-43a7-43fa-b8bd-5851d4679a19\") " pod="kuadrant-system/authorino-6d4dfc649f-sswmp"
Sep 11 16:10:58.848501 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.848385    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmz72\" (UniqueName: \"kubernetes.io/projected/f163a5d6-43a7-43fa-b8bd-5851d4679a19-kube-api-access-gmz72\") pod \"authorino-6d4dfc649f-sswmp\" (UID: \"f163a5d6-43a7-43fa-b8bd-5851d4679a19\") " pod="kuadrant-system/authorino-6d4dfc649f-sswmp"
Sep 11 16:10:58.850962 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.850939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f163a5d6-43a7-43fa-b8bd-5851d4679a19-tls-cert\") pod \"authorino-6d4dfc649f-sswmp\" (UID: \"f163a5d6-43a7-43fa-b8bd-5851d4679a19\") " pod="kuadrant-system/authorino-6d4dfc649f-sswmp"
Sep 11 16:10:58.864234 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.864206    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c75d4b78-jpq7m"]
Sep 11 16:10:58.867983 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.867966    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c75d4b78-jpq7m"
Sep 11 16:10:58.879834 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.879808    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmz72\" (UniqueName: \"kubernetes.io/projected/f163a5d6-43a7-43fa-b8bd-5851d4679a19-kube-api-access-gmz72\") pod \"authorino-6d4dfc649f-sswmp\" (UID: \"f163a5d6-43a7-43fa-b8bd-5851d4679a19\") " pod="kuadrant-system/authorino-6d4dfc649f-sswmp"
Sep 11 16:10:58.911602 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.911573    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c75d4b78-jpq7m"]
Sep 11 16:10:58.948810 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.948783    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f163a5d6-43a7-43fa-b8bd-5851d4679a19-tls-cert\") pod \"f163a5d6-43a7-43fa-b8bd-5851d4679a19\" (UID: \"f163a5d6-43a7-43fa-b8bd-5851d4679a19\") "
Sep 11 16:10:58.948995 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.948848    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7llb5\" (UniqueName: \"kubernetes.io/projected/14bb3596-4768-4a41-aa88-6bd2c58ec8fa-kube-api-access-7llb5\") pod \"14bb3596-4768-4a41-aa88-6bd2c58ec8fa\" (UID: \"14bb3596-4768-4a41-aa88-6bd2c58ec8fa\") "
Sep 11 16:10:58.948995 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.948933    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e33120ab-3871-462f-af72-18f57df9d160-tls-cert\") pod \"authorino-59c75d4b78-jpq7m\" (UID: \"e33120ab-3871-462f-af72-18f57df9d160\") " pod="kuadrant-system/authorino-59c75d4b78-jpq7m"
Sep 11 16:10:58.948995 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.948966    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w4j2\" (UniqueName: \"kubernetes.io/projected/e33120ab-3871-462f-af72-18f57df9d160-kube-api-access-4w4j2\") pod \"authorino-59c75d4b78-jpq7m\" (UID: \"e33120ab-3871-462f-af72-18f57df9d160\") " pod="kuadrant-system/authorino-59c75d4b78-jpq7m"
Sep 11 16:10:58.951062 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.951027    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f163a5d6-43a7-43fa-b8bd-5851d4679a19-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f163a5d6-43a7-43fa-b8bd-5851d4679a19" (UID: "f163a5d6-43a7-43fa-b8bd-5851d4679a19"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:10:58.951197 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:58.951174    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14bb3596-4768-4a41-aa88-6bd2c58ec8fa-kube-api-access-7llb5" (OuterVolumeSpecName: "kube-api-access-7llb5") pod "14bb3596-4768-4a41-aa88-6bd2c58ec8fa" (UID: "14bb3596-4768-4a41-aa88-6bd2c58ec8fa"). InnerVolumeSpecName "kube-api-access-7llb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:10:59.049590 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:59.049512    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmz72\" (UniqueName: \"kubernetes.io/projected/f163a5d6-43a7-43fa-b8bd-5851d4679a19-kube-api-access-gmz72\") pod \"f163a5d6-43a7-43fa-b8bd-5851d4679a19\" (UID: \"f163a5d6-43a7-43fa-b8bd-5851d4679a19\") "
Sep 11 16:10:59.049735 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:59.049633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e33120ab-3871-462f-af72-18f57df9d160-tls-cert\") pod \"authorino-59c75d4b78-jpq7m\" (UID: \"e33120ab-3871-462f-af72-18f57df9d160\") " pod="kuadrant-system/authorino-59c75d4b78-jpq7m"
Sep 11 16:10:59.049735 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:59.049667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w4j2\" (UniqueName: \"kubernetes.io/projected/e33120ab-3871-462f-af72-18f57df9d160-kube-api-access-4w4j2\") pod \"authorino-59c75d4b78-jpq7m\" (UID: \"e33120ab-3871-462f-af72-18f57df9d160\") " pod="kuadrant-system/authorino-59c75d4b78-jpq7m"
Sep 11 16:10:59.049735 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:59.049719    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f163a5d6-43a7-43fa-b8bd-5851d4679a19-tls-cert\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:10:59.049735 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:59.049729    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7llb5\" (UniqueName: \"kubernetes.io/projected/14bb3596-4768-4a41-aa88-6bd2c58ec8fa-kube-api-access-7llb5\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:10:59.051753 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:59.051725    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f163a5d6-43a7-43fa-b8bd-5851d4679a19-kube-api-access-gmz72" (OuterVolumeSpecName: "kube-api-access-gmz72") pod "f163a5d6-43a7-43fa-b8bd-5851d4679a19" (UID: "f163a5d6-43a7-43fa-b8bd-5851d4679a19"). InnerVolumeSpecName "kube-api-access-gmz72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:10:59.052119 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:59.052103    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e33120ab-3871-462f-af72-18f57df9d160-tls-cert\") pod \"authorino-59c75d4b78-jpq7m\" (UID: \"e33120ab-3871-462f-af72-18f57df9d160\") " pod="kuadrant-system/authorino-59c75d4b78-jpq7m"
Sep 11 16:10:59.064844 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:59.064813    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w4j2\" (UniqueName: \"kubernetes.io/projected/e33120ab-3871-462f-af72-18f57df9d160-kube-api-access-4w4j2\") pod \"authorino-59c75d4b78-jpq7m\" (UID: \"e33120ab-3871-462f-af72-18f57df9d160\") " pod="kuadrant-system/authorino-59c75d4b78-jpq7m"
Sep 11 16:10:59.151096 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:59.151056    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmz72\" (UniqueName: \"kubernetes.io/projected/f163a5d6-43a7-43fa-b8bd-5851d4679a19-kube-api-access-gmz72\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:10:59.178150 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:59.178115    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c75d4b78-jpq7m"
Sep 11 16:10:59.461143 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:10:59.461102    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode33120ab_3871_462f_af72_18f57df9d160.slice/crio-9a9a27eed0945a638c8d4b87783623d326d40e4c1c88e718f183a73bda4d8a37 WatchSource:0}: Error finding container 9a9a27eed0945a638c8d4b87783623d326d40e4c1c88e718f183a73bda4d8a37: Status 404 returned error can't find the container with id 9a9a27eed0945a638c8d4b87783623d326d40e4c1c88e718f183a73bda4d8a37
Sep 11 16:10:59.461724 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:59.461703    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c75d4b78-jpq7m"]
Sep 11 16:10:59.806400 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:59.806314    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-vrxgr"
Sep 11 16:10:59.806400 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:59.806343    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c75d4b78-jpq7m" event={"ID":"e33120ab-3871-462f-af72-18f57df9d160","Type":"ContainerStarted","Data":"9a9a27eed0945a638c8d4b87783623d326d40e4c1c88e718f183a73bda4d8a37"}
Sep 11 16:10:59.806586 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:59.806489    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-sswmp"
Sep 11 16:10:59.850197 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:59.850156    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-vrxgr"]
Sep 11 16:10:59.852758 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:59.852735    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-vrxgr"]
Sep 11 16:10:59.884438 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:59.884407    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-sswmp"]
Sep 11 16:10:59.891929 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:10:59.891903    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-sswmp"]
Sep 11 16:11:00.632817 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:00.632774    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14bb3596-4768-4a41-aa88-6bd2c58ec8fa" path="/var/lib/kubelet/pods/14bb3596-4768-4a41-aa88-6bd2c58ec8fa/volumes"
Sep 11 16:11:00.633114 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:00.633097    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f163a5d6-43a7-43fa-b8bd-5851d4679a19" path="/var/lib/kubelet/pods/f163a5d6-43a7-43fa-b8bd-5851d4679a19/volumes"
Sep 11 16:11:00.812981 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:00.812945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c75d4b78-jpq7m" event={"ID":"e33120ab-3871-462f-af72-18f57df9d160","Type":"ContainerStarted","Data":"a3e5ec3ed61be393ffc5f6e12967b100c1022669fa2584a9459c6385731a8dec"}
Sep 11 16:11:00.846339 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:00.846287    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c75d4b78-jpq7m" podStartSLOduration=2.240076975 podStartE2EDuration="2.846266681s" totalImagesPullingTime="606.189706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:11:00.845232186 +0000 UTC m=+520.690875739" watchObservedRunningTime="2026-09-11 16:11:00.846266681 +0000 UTC m=+520.691910239"
Sep 11 16:11:00.932680 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:00.932600    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-kqmgp"]
Sep 11 16:11:00.933099 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:00.932771    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-kqmgp" podUID="f393a664-8c41-41d8-98d5-cd1d120f8f31" containerName="authorino" containerID="cri-o://42eef37ac91610fb0a5f6113c1132c8959fc54299b462307691fdb596c6d1cd4" gracePeriod=30
Sep 11 16:11:01.200843 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:01.200766    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-kqmgp"
Sep 11 16:11:01.370322 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:01.370280    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgztj\" (UniqueName: \"kubernetes.io/projected/f393a664-8c41-41d8-98d5-cd1d120f8f31-kube-api-access-jgztj\") pod \"f393a664-8c41-41d8-98d5-cd1d120f8f31\" (UID: \"f393a664-8c41-41d8-98d5-cd1d120f8f31\") "
Sep 11 16:11:01.372432 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:01.372405    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f393a664-8c41-41d8-98d5-cd1d120f8f31-kube-api-access-jgztj" (OuterVolumeSpecName: "kube-api-access-jgztj") pod "f393a664-8c41-41d8-98d5-cd1d120f8f31" (UID: "f393a664-8c41-41d8-98d5-cd1d120f8f31"). InnerVolumeSpecName "kube-api-access-jgztj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:11:01.471480 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:01.471387    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgztj\" (UniqueName: \"kubernetes.io/projected/f393a664-8c41-41d8-98d5-cd1d120f8f31-kube-api-access-jgztj\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:11:01.818826 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:01.818727    2577 generic.go:356] "Generic (PLEG): container finished" podID="f393a664-8c41-41d8-98d5-cd1d120f8f31" containerID="42eef37ac91610fb0a5f6113c1132c8959fc54299b462307691fdb596c6d1cd4" exitCode=0
Sep 11 16:11:01.818826 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:01.818767    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-kqmgp"
Sep 11 16:11:01.818826 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:01.818790    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-kqmgp" event={"ID":"f393a664-8c41-41d8-98d5-cd1d120f8f31","Type":"ContainerDied","Data":"42eef37ac91610fb0a5f6113c1132c8959fc54299b462307691fdb596c6d1cd4"}
Sep 11 16:11:01.819116 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:01.818832    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-kqmgp" event={"ID":"f393a664-8c41-41d8-98d5-cd1d120f8f31","Type":"ContainerDied","Data":"d6364a4afe33ae058f833321c7d53dc7b9b7e1724be210a040d5f9ff492683e5"}
Sep 11 16:11:01.819116 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:01.818853    2577 scope.go:117] "RemoveContainer" containerID="42eef37ac91610fb0a5f6113c1132c8959fc54299b462307691fdb596c6d1cd4"
Sep 11 16:11:01.828000 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:01.827980    2577 scope.go:117] "RemoveContainer" containerID="42eef37ac91610fb0a5f6113c1132c8959fc54299b462307691fdb596c6d1cd4"
Sep 11 16:11:01.828387 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:11:01.828360    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42eef37ac91610fb0a5f6113c1132c8959fc54299b462307691fdb596c6d1cd4\": container with ID starting with 42eef37ac91610fb0a5f6113c1132c8959fc54299b462307691fdb596c6d1cd4 not found: ID does not exist" containerID="42eef37ac91610fb0a5f6113c1132c8959fc54299b462307691fdb596c6d1cd4"
Sep 11 16:11:01.828439 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:01.828403    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42eef37ac91610fb0a5f6113c1132c8959fc54299b462307691fdb596c6d1cd4"} err="failed to get container status \"42eef37ac91610fb0a5f6113c1132c8959fc54299b462307691fdb596c6d1cd4\": rpc error: code = NotFound desc = could not find container \"42eef37ac91610fb0a5f6113c1132c8959fc54299b462307691fdb596c6d1cd4\": container with ID starting with 42eef37ac91610fb0a5f6113c1132c8959fc54299b462307691fdb596c6d1cd4 not found: ID does not exist"
Sep 11 16:11:01.847419 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:01.847388    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-kqmgp"]
Sep 11 16:11:01.851673 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:01.851644    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-kqmgp"]
Sep 11 16:11:01.852608 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:11:01.852585    2577 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf393a664_8c41_41d8_98d5_cd1d120f8f31.slice\": RecentStats: unable to find data in memory cache]"
Sep 11 16:11:02.631936 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:02.631901    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f393a664-8c41-41d8-98d5-cd1d120f8f31" path="/var/lib/kubelet/pods/f393a664-8c41-41d8-98d5-cd1d120f8f31/volumes"
Sep 11 16:11:06.003359 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.003325    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f79c8dc6c-cncpj"]
Sep 11 16:11:06.003801 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.003651    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f393a664-8c41-41d8-98d5-cd1d120f8f31" containerName="authorino"
Sep 11 16:11:06.003801 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.003662    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f393a664-8c41-41d8-98d5-cd1d120f8f31" containerName="authorino"
Sep 11 16:11:06.003801 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.003709    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f393a664-8c41-41d8-98d5-cd1d120f8f31" containerName="authorino"
Sep 11 16:11:06.017582 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.017547    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f79c8dc6c-cncpj"
Sep 11 16:11:06.021550 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.021520    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5frtc\""
Sep 11 16:11:06.021717 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.021554    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 16:11:06.021842 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.021822    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 16:11:06.022323 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.022301    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 16:11:06.022968 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.022942    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f79c8dc6c-cncpj"]
Sep 11 16:11:06.107482 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.107439    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5781f6-abb7-446b-9fdd-274af7985618-metrics-certs\") pod \"maas-controller-5f79c8dc6c-cncpj\" (UID: \"2e5781f6-abb7-446b-9fdd-274af7985618\") " pod="opendatahub/maas-controller-5f79c8dc6c-cncpj"
Sep 11 16:11:06.107482 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.107487    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e5781f6-abb7-446b-9fdd-274af7985618-webhook-cert\") pod \"maas-controller-5f79c8dc6c-cncpj\" (UID: \"2e5781f6-abb7-446b-9fdd-274af7985618\") " pod="opendatahub/maas-controller-5f79c8dc6c-cncpj"
Sep 11 16:11:06.107707 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.107507    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6cb8\" (UniqueName: \"kubernetes.io/projected/2e5781f6-abb7-446b-9fdd-274af7985618-kube-api-access-v6cb8\") pod \"maas-controller-5f79c8dc6c-cncpj\" (UID: \"2e5781f6-abb7-446b-9fdd-274af7985618\") " pod="opendatahub/maas-controller-5f79c8dc6c-cncpj"
Sep 11 16:11:06.208640 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.208605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5781f6-abb7-446b-9fdd-274af7985618-metrics-certs\") pod \"maas-controller-5f79c8dc6c-cncpj\" (UID: \"2e5781f6-abb7-446b-9fdd-274af7985618\") " pod="opendatahub/maas-controller-5f79c8dc6c-cncpj"
Sep 11 16:11:06.208849 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.208647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e5781f6-abb7-446b-9fdd-274af7985618-webhook-cert\") pod \"maas-controller-5f79c8dc6c-cncpj\" (UID: \"2e5781f6-abb7-446b-9fdd-274af7985618\") " pod="opendatahub/maas-controller-5f79c8dc6c-cncpj"
Sep 11 16:11:06.208849 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.208669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6cb8\" (UniqueName: \"kubernetes.io/projected/2e5781f6-abb7-446b-9fdd-274af7985618-kube-api-access-v6cb8\") pod \"maas-controller-5f79c8dc6c-cncpj\" (UID: \"2e5781f6-abb7-446b-9fdd-274af7985618\") " pod="opendatahub/maas-controller-5f79c8dc6c-cncpj"
Sep 11 16:11:06.208849 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:11:06.208783    2577 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 11 16:11:06.209030 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:11:06.208863    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e5781f6-abb7-446b-9fdd-274af7985618-webhook-cert podName:2e5781f6-abb7-446b-9fdd-274af7985618 nodeName:}" failed. No retries permitted until 2026-09-11 16:11:06.708839079 +0000 UTC m=+526.554482629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/2e5781f6-abb7-446b-9fdd-274af7985618-webhook-cert") pod "maas-controller-5f79c8dc6c-cncpj" (UID: "2e5781f6-abb7-446b-9fdd-274af7985618") : secret "maas-controller-webhook-cert" not found
Sep 11 16:11:06.209030 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:11:06.208783    2577 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 11 16:11:06.209030 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:11:06.208943    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e5781f6-abb7-446b-9fdd-274af7985618-metrics-certs podName:2e5781f6-abb7-446b-9fdd-274af7985618 nodeName:}" failed. No retries permitted until 2026-09-11 16:11:06.708923955 +0000 UTC m=+526.554567490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e5781f6-abb7-446b-9fdd-274af7985618-metrics-certs") pod "maas-controller-5f79c8dc6c-cncpj" (UID: "2e5781f6-abb7-446b-9fdd-274af7985618") : secret "maas-controller-metrics-tls" not found
Sep 11 16:11:06.240027 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.239996    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6cb8\" (UniqueName: \"kubernetes.io/projected/2e5781f6-abb7-446b-9fdd-274af7985618-kube-api-access-v6cb8\") pod \"maas-controller-5f79c8dc6c-cncpj\" (UID: \"2e5781f6-abb7-446b-9fdd-274af7985618\") " pod="opendatahub/maas-controller-5f79c8dc6c-cncpj"
Sep 11 16:11:06.712732 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.712692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e5781f6-abb7-446b-9fdd-274af7985618-webhook-cert\") pod \"maas-controller-5f79c8dc6c-cncpj\" (UID: \"2e5781f6-abb7-446b-9fdd-274af7985618\") " pod="opendatahub/maas-controller-5f79c8dc6c-cncpj"
Sep 11 16:11:06.712934 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.712787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5781f6-abb7-446b-9fdd-274af7985618-metrics-certs\") pod \"maas-controller-5f79c8dc6c-cncpj\" (UID: \"2e5781f6-abb7-446b-9fdd-274af7985618\") " pod="opendatahub/maas-controller-5f79c8dc6c-cncpj"
Sep 11 16:11:06.715304 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.715263    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5781f6-abb7-446b-9fdd-274af7985618-metrics-certs\") pod \"maas-controller-5f79c8dc6c-cncpj\" (UID: \"2e5781f6-abb7-446b-9fdd-274af7985618\") " pod="opendatahub/maas-controller-5f79c8dc6c-cncpj"
Sep 11 16:11:06.715441 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.715361    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e5781f6-abb7-446b-9fdd-274af7985618-webhook-cert\") pod \"maas-controller-5f79c8dc6c-cncpj\" (UID: \"2e5781f6-abb7-446b-9fdd-274af7985618\") " pod="opendatahub/maas-controller-5f79c8dc6c-cncpj"
Sep 11 16:11:06.931399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:06.931365    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f79c8dc6c-cncpj"
Sep 11 16:11:07.209177 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:07.209134    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f79c8dc6c-cncpj"]
Sep 11 16:11:07.212920 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:11:07.212894    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5781f6_abb7_446b_9fdd_274af7985618.slice/crio-01ce750e858ea182bb51d5e2b048ac9b80b9a66e498b5b77fb429bc38b70f548 WatchSource:0}: Error finding container 01ce750e858ea182bb51d5e2b048ac9b80b9a66e498b5b77fb429bc38b70f548: Status 404 returned error can't find the container with id 01ce750e858ea182bb51d5e2b048ac9b80b9a66e498b5b77fb429bc38b70f548
Sep 11 16:11:07.854084 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:07.854018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f79c8dc6c-cncpj" event={"ID":"2e5781f6-abb7-446b-9fdd-274af7985618","Type":"ContainerStarted","Data":"01ce750e858ea182bb51d5e2b048ac9b80b9a66e498b5b77fb429bc38b70f548"}
Sep 11 16:11:09.866524 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:09.866489    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f79c8dc6c-cncpj" event={"ID":"2e5781f6-abb7-446b-9fdd-274af7985618","Type":"ContainerStarted","Data":"a02b0d3b1532e506beaf9c59d22b3e782e21d7db6d4bac69fe9cfc4ba7b4254d"}
Sep 11 16:11:09.866928 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:09.866614    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f79c8dc6c-cncpj"
Sep 11 16:11:09.892108 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:09.891874    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5f79c8dc6c-cncpj" podStartSLOduration=2.4624552250000002 podStartE2EDuration="4.891854158s" totalImagesPullingTime="2.429398933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:11:09.890825476 +0000 UTC m=+529.736469030" watchObservedRunningTime="2026-09-11 16:11:09.891854158 +0000 UTC m=+529.737497714"
Sep 11 16:11:20.873637 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:20.873608    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f79c8dc6c-cncpj"
Sep 11 16:11:21.252800 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.252719    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f79c8dc6c-cncpj"]
Sep 11 16:11:21.252947 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.252917    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5f79c8dc6c-cncpj" podUID="2e5781f6-abb7-446b-9fdd-274af7985618" containerName="manager" containerID="cri-o://a02b0d3b1532e506beaf9c59d22b3e782e21d7db6d4bac69fe9cfc4ba7b4254d" gracePeriod=30
Sep 11 16:11:21.497029 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.497000    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f79c8dc6c-cncpj"
Sep 11 16:11:21.522477 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.522385    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6cb8\" (UniqueName: \"kubernetes.io/projected/2e5781f6-abb7-446b-9fdd-274af7985618-kube-api-access-v6cb8\") pod \"2e5781f6-abb7-446b-9fdd-274af7985618\" (UID: \"2e5781f6-abb7-446b-9fdd-274af7985618\") "
Sep 11 16:11:21.522477 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.522441    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5781f6-abb7-446b-9fdd-274af7985618-metrics-certs\") pod \"2e5781f6-abb7-446b-9fdd-274af7985618\" (UID: \"2e5781f6-abb7-446b-9fdd-274af7985618\") "
Sep 11 16:11:21.524956 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.524923    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5781f6-abb7-446b-9fdd-274af7985618-kube-api-access-v6cb8" (OuterVolumeSpecName: "kube-api-access-v6cb8") pod "2e5781f6-abb7-446b-9fdd-274af7985618" (UID: "2e5781f6-abb7-446b-9fdd-274af7985618"). InnerVolumeSpecName "kube-api-access-v6cb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:11:21.525142 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.525026    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e5781f6-abb7-446b-9fdd-274af7985618-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2e5781f6-abb7-446b-9fdd-274af7985618" (UID: "2e5781f6-abb7-446b-9fdd-274af7985618"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:11:21.623148 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.623108    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e5781f6-abb7-446b-9fdd-274af7985618-webhook-cert\") pod \"2e5781f6-abb7-446b-9fdd-274af7985618\" (UID: \"2e5781f6-abb7-446b-9fdd-274af7985618\") "
Sep 11 16:11:21.623312 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.623299    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6cb8\" (UniqueName: \"kubernetes.io/projected/2e5781f6-abb7-446b-9fdd-274af7985618-kube-api-access-v6cb8\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:11:21.623370 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.623315    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5781f6-abb7-446b-9fdd-274af7985618-metrics-certs\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:11:21.625248 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.625213    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e5781f6-abb7-446b-9fdd-274af7985618-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2e5781f6-abb7-446b-9fdd-274af7985618" (UID: "2e5781f6-abb7-446b-9fdd-274af7985618"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:11:21.723854 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.723806    2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e5781f6-abb7-446b-9fdd-274af7985618-webhook-cert\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:11:21.923146 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.923107    2577 generic.go:356] "Generic (PLEG): container finished" podID="2e5781f6-abb7-446b-9fdd-274af7985618" containerID="a02b0d3b1532e506beaf9c59d22b3e782e21d7db6d4bac69fe9cfc4ba7b4254d" exitCode=0
Sep 11 16:11:21.923146 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.923152    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f79c8dc6c-cncpj"
Sep 11 16:11:21.923589 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.923186    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f79c8dc6c-cncpj" event={"ID":"2e5781f6-abb7-446b-9fdd-274af7985618","Type":"ContainerDied","Data":"a02b0d3b1532e506beaf9c59d22b3e782e21d7db6d4bac69fe9cfc4ba7b4254d"}
Sep 11 16:11:21.923589 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.923238    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f79c8dc6c-cncpj" event={"ID":"2e5781f6-abb7-446b-9fdd-274af7985618","Type":"ContainerDied","Data":"01ce750e858ea182bb51d5e2b048ac9b80b9a66e498b5b77fb429bc38b70f548"}
Sep 11 16:11:21.923589 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.923260    2577 scope.go:117] "RemoveContainer" containerID="a02b0d3b1532e506beaf9c59d22b3e782e21d7db6d4bac69fe9cfc4ba7b4254d"
Sep 11 16:11:21.932137 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.932114    2577 scope.go:117] "RemoveContainer" containerID="a02b0d3b1532e506beaf9c59d22b3e782e21d7db6d4bac69fe9cfc4ba7b4254d"
Sep 11 16:11:21.932398 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:11:21.932379    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a02b0d3b1532e506beaf9c59d22b3e782e21d7db6d4bac69fe9cfc4ba7b4254d\": container with ID starting with a02b0d3b1532e506beaf9c59d22b3e782e21d7db6d4bac69fe9cfc4ba7b4254d not found: ID does not exist" containerID="a02b0d3b1532e506beaf9c59d22b3e782e21d7db6d4bac69fe9cfc4ba7b4254d"
Sep 11 16:11:21.932448 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.932408    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02b0d3b1532e506beaf9c59d22b3e782e21d7db6d4bac69fe9cfc4ba7b4254d"} err="failed to get container status \"a02b0d3b1532e506beaf9c59d22b3e782e21d7db6d4bac69fe9cfc4ba7b4254d\": rpc error: code = NotFound desc = could not find container \"a02b0d3b1532e506beaf9c59d22b3e782e21d7db6d4bac69fe9cfc4ba7b4254d\": container with ID starting with a02b0d3b1532e506beaf9c59d22b3e782e21d7db6d4bac69fe9cfc4ba7b4254d not found: ID does not exist"
Sep 11 16:11:21.947787 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.947762    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f79c8dc6c-cncpj"]
Sep 11 16:11:21.953768 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:21.953747    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5f79c8dc6c-cncpj"]
Sep 11 16:11:22.634448 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:22.634414    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e5781f6-abb7-446b-9fdd-274af7985618" path="/var/lib/kubelet/pods/2e5781f6-abb7-446b-9fdd-274af7985618/volumes"
Sep 11 16:11:42.399317 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:42.399285    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-845964844c-8cfwp"]
Sep 11 16:11:42.399865 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:42.399709    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e5781f6-abb7-446b-9fdd-274af7985618" containerName="manager"
Sep 11 16:11:42.399865 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:42.399728    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e5781f6-abb7-446b-9fdd-274af7985618" containerName="manager"
Sep 11 16:11:42.399865 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:42.399810    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e5781f6-abb7-446b-9fdd-274af7985618" containerName="manager"
Sep 11 16:11:42.405931 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:42.405914    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp"
Sep 11 16:11:42.408955 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:42.408923    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p2kww\""
Sep 11 16:11:42.409243 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:42.409226    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 16:11:42.409325 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:42.409235    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 16:11:42.414382 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:42.414359    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-845964844c-8cfwp"]
Sep 11 16:11:42.477209 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:42.477178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c138bc01-6529-4072-813f-d5102dca4d1b-metrics-certs\") pod \"maas-api-845964844c-8cfwp\" (UID: \"c138bc01-6529-4072-813f-d5102dca4d1b\") " pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp"
Sep 11 16:11:42.477383 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:42.477228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c138bc01-6529-4072-813f-d5102dca4d1b-openshift-service-ca\") pod \"maas-api-845964844c-8cfwp\" (UID: \"c138bc01-6529-4072-813f-d5102dca4d1b\") " pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp"
Sep 11 16:11:42.477383 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:42.477248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c138bc01-6529-4072-813f-d5102dca4d1b-maas-api-tls\") pod \"maas-api-845964844c-8cfwp\" (UID: \"c138bc01-6529-4072-813f-d5102dca4d1b\") " pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp"
Sep 11 16:11:42.477383 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:42.477311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24g8r\" (UniqueName: \"kubernetes.io/projected/c138bc01-6529-4072-813f-d5102dca4d1b-kube-api-access-24g8r\") pod \"maas-api-845964844c-8cfwp\" (UID: \"c138bc01-6529-4072-813f-d5102dca4d1b\") " pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp"
Sep 11 16:11:42.578626 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:42.578592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c138bc01-6529-4072-813f-d5102dca4d1b-openshift-service-ca\") pod \"maas-api-845964844c-8cfwp\" (UID: \"c138bc01-6529-4072-813f-d5102dca4d1b\") " pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp"
Sep 11 16:11:42.578626 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:42.578627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c138bc01-6529-4072-813f-d5102dca4d1b-maas-api-tls\") pod \"maas-api-845964844c-8cfwp\" (UID: \"c138bc01-6529-4072-813f-d5102dca4d1b\") " pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp"
Sep 11 16:11:42.578883 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:42.578652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24g8r\" (UniqueName: \"kubernetes.io/projected/c138bc01-6529-4072-813f-d5102dca4d1b-kube-api-access-24g8r\") pod \"maas-api-845964844c-8cfwp\" (UID: \"c138bc01-6529-4072-813f-d5102dca4d1b\") " pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp"
Sep 11 16:11:42.578883 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:42.578726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c138bc01-6529-4072-813f-d5102dca4d1b-metrics-certs\") pod \"maas-api-845964844c-8cfwp\" (UID: \"c138bc01-6529-4072-813f-d5102dca4d1b\") " pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp"
Sep 11 16:11:42.578883 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:11:42.578737    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 11 16:11:42.578883 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:11:42.578860    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c138bc01-6529-4072-813f-d5102dca4d1b-maas-api-tls podName:c138bc01-6529-4072-813f-d5102dca4d1b nodeName:}" failed. No retries permitted until 2026-09-11 16:11:43.0788351 +0000 UTC m=+562.924478646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c138bc01-6529-4072-813f-d5102dca4d1b-maas-api-tls") pod "maas-api-845964844c-8cfwp" (UID: "c138bc01-6529-4072-813f-d5102dca4d1b") : secret "maas-api-serving-cert" not found
Sep 11 16:11:42.578883 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:11:42.578863    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 11 16:11:42.579178 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:11:42.578907    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c138bc01-6529-4072-813f-d5102dca4d1b-metrics-certs podName:c138bc01-6529-4072-813f-d5102dca4d1b nodeName:}" failed. No retries permitted until 2026-09-11 16:11:43.078893406 +0000 UTC m=+562.924536939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c138bc01-6529-4072-813f-d5102dca4d1b-metrics-certs") pod "maas-api-845964844c-8cfwp" (UID: "c138bc01-6529-4072-813f-d5102dca4d1b") : secret "maas-api-metrics-tls" not found
Sep 11 16:11:42.579331 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:42.579312    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c138bc01-6529-4072-813f-d5102dca4d1b-openshift-service-ca\") pod \"maas-api-845964844c-8cfwp\" (UID: \"c138bc01-6529-4072-813f-d5102dca4d1b\") " pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp"
Sep 11 16:11:42.588799 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:42.588777    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24g8r\" (UniqueName: \"kubernetes.io/projected/c138bc01-6529-4072-813f-d5102dca4d1b-kube-api-access-24g8r\") pod \"maas-api-845964844c-8cfwp\" (UID: \"c138bc01-6529-4072-813f-d5102dca4d1b\") " pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp"
Sep 11 16:11:43.049815 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:43.049783    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fl68q"]
Sep 11 16:11:43.053137 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:43.053117    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fl68q"
Sep 11 16:11:43.055714 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:43.055691    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 16:11:43.055820 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:43.055772    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l865n\""
Sep 11 16:11:43.064704 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:43.064676    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fl68q"]
Sep 11 16:11:43.082426 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:43.082397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnkfs\" (UniqueName: \"kubernetes.io/projected/77c07ed6-da3e-4ab4-a7d7-ba7de106c633-kube-api-access-wnkfs\") pod \"payload-pre-processing-59fbc47bf8-fl68q\" (UID: \"77c07ed6-da3e-4ab4-a7d7-ba7de106c633\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fl68q"
Sep 11 16:11:43.082576 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:43.082448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c138bc01-6529-4072-813f-d5102dca4d1b-maas-api-tls\") pod \"maas-api-845964844c-8cfwp\" (UID: \"c138bc01-6529-4072-813f-d5102dca4d1b\") " pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp"
Sep 11 16:11:43.082576 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:43.082501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c07ed6-da3e-4ab4-a7d7-ba7de106c633-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fl68q\" (UID: \"77c07ed6-da3e-4ab4-a7d7-ba7de106c633\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fl68q"
Sep 11 16:11:43.082576 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:43.082520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c138bc01-6529-4072-813f-d5102dca4d1b-metrics-certs\") pod \"maas-api-845964844c-8cfwp\" (UID: \"c138bc01-6529-4072-813f-d5102dca4d1b\") " pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp"
Sep 11 16:11:43.082735 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:11:43.082615    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 11 16:11:43.082735 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:11:43.082676    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c138bc01-6529-4072-813f-d5102dca4d1b-metrics-certs podName:c138bc01-6529-4072-813f-d5102dca4d1b nodeName:}" failed. No retries permitted until 2026-09-11 16:11:44.082659633 +0000 UTC m=+563.928303167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c138bc01-6529-4072-813f-d5102dca4d1b-metrics-certs") pod "maas-api-845964844c-8cfwp" (UID: "c138bc01-6529-4072-813f-d5102dca4d1b") : secret "maas-api-metrics-tls" not found
Sep 11 16:11:43.085280 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:43.085255    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c138bc01-6529-4072-813f-d5102dca4d1b-maas-api-tls\") pod \"maas-api-845964844c-8cfwp\" (UID: \"c138bc01-6529-4072-813f-d5102dca4d1b\") " pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp"
Sep 11 16:11:43.183104 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:43.183074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnkfs\" (UniqueName: \"kubernetes.io/projected/77c07ed6-da3e-4ab4-a7d7-ba7de106c633-kube-api-access-wnkfs\") pod \"payload-pre-processing-59fbc47bf8-fl68q\" (UID: \"77c07ed6-da3e-4ab4-a7d7-ba7de106c633\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fl68q"
Sep 11 16:11:43.183296 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:43.183162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c07ed6-da3e-4ab4-a7d7-ba7de106c633-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fl68q\" (UID: \"77c07ed6-da3e-4ab4-a7d7-ba7de106c633\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fl68q"
Sep 11 16:11:43.183898 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:43.183879    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c07ed6-da3e-4ab4-a7d7-ba7de106c633-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fl68q\" (UID: \"77c07ed6-da3e-4ab4-a7d7-ba7de106c633\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fl68q"
Sep 11 16:11:43.201409 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:43.201373    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnkfs\" (UniqueName: \"kubernetes.io/projected/77c07ed6-da3e-4ab4-a7d7-ba7de106c633-kube-api-access-wnkfs\") pod \"payload-pre-processing-59fbc47bf8-fl68q\" (UID: \"77c07ed6-da3e-4ab4-a7d7-ba7de106c633\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fl68q"
Sep 11 16:11:43.364049 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:43.363990    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fl68q"
Sep 11 16:11:43.661582 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:43.661554    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fl68q"]
Sep 11 16:11:43.663106 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:11:43.663079    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c07ed6_da3e_4ab4_a7d7_ba7de106c633.slice/crio-d0663f3c3a487254511ba5b8678736b751d1f5aad85aec588eaf3250c10a58bd WatchSource:0}: Error finding container d0663f3c3a487254511ba5b8678736b751d1f5aad85aec588eaf3250c10a58bd: Status 404 returned error can't find the container with id d0663f3c3a487254511ba5b8678736b751d1f5aad85aec588eaf3250c10a58bd
Sep 11 16:11:44.022995 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:44.022908    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fl68q" event={"ID":"77c07ed6-da3e-4ab4-a7d7-ba7de106c633","Type":"ContainerStarted","Data":"d0663f3c3a487254511ba5b8678736b751d1f5aad85aec588eaf3250c10a58bd"}
Sep 11 16:11:44.091589 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:44.091553    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c138bc01-6529-4072-813f-d5102dca4d1b-metrics-certs\") pod \"maas-api-845964844c-8cfwp\" (UID: \"c138bc01-6529-4072-813f-d5102dca4d1b\") " pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp"
Sep 11 16:11:44.094105 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:44.094081    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c138bc01-6529-4072-813f-d5102dca4d1b-metrics-certs\") pod \"maas-api-845964844c-8cfwp\" (UID: \"c138bc01-6529-4072-813f-d5102dca4d1b\") " pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp"
Sep 11 16:11:44.216724 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:44.216689    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp"
Sep 11 16:11:44.502071 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:44.501895    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-845964844c-8cfwp"]
Sep 11 16:11:44.505701 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:11:44.505668    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc138bc01_6529_4072_813f_d5102dca4d1b.slice/crio-f641978ff7cf80d0a3b49d306a9b7f0cbd82238cad9306ba6113f0022cb43741 WatchSource:0}: Error finding container f641978ff7cf80d0a3b49d306a9b7f0cbd82238cad9306ba6113f0022cb43741: Status 404 returned error can't find the container with id f641978ff7cf80d0a3b49d306a9b7f0cbd82238cad9306ba6113f0022cb43741
Sep 11 16:11:45.030444 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:45.030401    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp" event={"ID":"c138bc01-6529-4072-813f-d5102dca4d1b","Type":"ContainerStarted","Data":"f641978ff7cf80d0a3b49d306a9b7f0cbd82238cad9306ba6113f0022cb43741"}
Sep 11 16:11:46.042195 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:46.042151    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fl68q" event={"ID":"77c07ed6-da3e-4ab4-a7d7-ba7de106c633","Type":"ContainerStarted","Data":"11f874a03ad9f09c33dad97828ee7196f2e89aa6b23d7426b3e82fb397be8dac"}
Sep 11 16:11:46.042682 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:46.042446    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fl68q"
Sep 11 16:11:46.073193 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:46.073125    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fl68q" podStartSLOduration=1.274312119 podStartE2EDuration="3.073099163s" totalImagesPullingTime="1.798787044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:11:46.071533226 +0000 UTC m=+565.917176788" watchObservedRunningTime="2026-09-11 16:11:46.073099163 +0000 UTC m=+565.918742726"
Sep 11 16:11:47.048268 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:47.048228    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp" event={"ID":"c138bc01-6529-4072-813f-d5102dca4d1b","Type":"ContainerStarted","Data":"ebb22c1ba0fe831cdf74f5e6797c02211c7287948ea6454d3a169ea120ed3f3a"}
Sep 11 16:11:47.048737 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:47.048424    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp"
Sep 11 16:11:47.071941 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:47.071891    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp" podStartSLOduration=3.437811897 podStartE2EDuration="5.071875741s" totalImagesPullingTime="1.634063844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:11:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:11:47.070769237 +0000 UTC m=+566.916412795" watchObservedRunningTime="2026-09-11 16:11:47.071875741 +0000 UTC m=+566.917519296"
Sep 11 16:11:53.058479 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:53.058444    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-845964844c-8cfwp"
Sep 11 16:11:57.050473 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:11:57.050444    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fl68q"
Sep 11 16:12:20.533919 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:12:20.533891    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/1.log"
Sep 11 16:12:20.535280 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:12:20.535257    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/1.log"
Sep 11 16:12:20.541065 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:12:20.541027    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ng2ww_07d57c04-0697-419b-9d3e-ca11761da85c/ovn-acl-logging/0.log"
Sep 11 16:12:20.542327 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:12:20.542312    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ng2ww_07d57c04-0697-419b-9d3e-ca11761da85c/ovn-acl-logging/0.log"
Sep 11 16:13:04.272613 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.272536    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"]
Sep 11 16:13:04.276135 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.276105    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:04.280250 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.280228    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 16:13:04.281232 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.281193    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zrm4z\""
Sep 11 16:13:04.281232 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.281216    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 11 16:13:04.282065 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.282048    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 16:13:04.341055 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.340997    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"]
Sep 11 16:13:04.360787 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.360744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw2bd\" (UniqueName: \"kubernetes.io/projected/fb15cdc1-cc17-4c1b-bec5-b757c56bcef8-kube-api-access-hw2bd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc\" (UID: \"fb15cdc1-cc17-4c1b-bec5-b757c56bcef8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:04.360955 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.360798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb15cdc1-cc17-4c1b-bec5-b757c56bcef8-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc\" (UID: \"fb15cdc1-cc17-4c1b-bec5-b757c56bcef8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:04.360955 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.360848    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb15cdc1-cc17-4c1b-bec5-b757c56bcef8-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc\" (UID: \"fb15cdc1-cc17-4c1b-bec5-b757c56bcef8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:04.360955 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.360892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb15cdc1-cc17-4c1b-bec5-b757c56bcef8-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc\" (UID: \"fb15cdc1-cc17-4c1b-bec5-b757c56bcef8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:04.360955 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.360940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb15cdc1-cc17-4c1b-bec5-b757c56bcef8-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc\" (UID: \"fb15cdc1-cc17-4c1b-bec5-b757c56bcef8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:04.361152 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.360964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb15cdc1-cc17-4c1b-bec5-b757c56bcef8-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc\" (UID: \"fb15cdc1-cc17-4c1b-bec5-b757c56bcef8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:04.461698 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.461653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw2bd\" (UniqueName: \"kubernetes.io/projected/fb15cdc1-cc17-4c1b-bec5-b757c56bcef8-kube-api-access-hw2bd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc\" (UID: \"fb15cdc1-cc17-4c1b-bec5-b757c56bcef8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:04.461698 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.461702    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb15cdc1-cc17-4c1b-bec5-b757c56bcef8-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc\" (UID: \"fb15cdc1-cc17-4c1b-bec5-b757c56bcef8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:04.461974 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.461727    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb15cdc1-cc17-4c1b-bec5-b757c56bcef8-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc\" (UID: \"fb15cdc1-cc17-4c1b-bec5-b757c56bcef8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:04.461974 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.461742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb15cdc1-cc17-4c1b-bec5-b757c56bcef8-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc\" (UID: \"fb15cdc1-cc17-4c1b-bec5-b757c56bcef8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:04.461974 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.461779    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb15cdc1-cc17-4c1b-bec5-b757c56bcef8-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc\" (UID: \"fb15cdc1-cc17-4c1b-bec5-b757c56bcef8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:04.461974 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.461908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb15cdc1-cc17-4c1b-bec5-b757c56bcef8-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc\" (UID: \"fb15cdc1-cc17-4c1b-bec5-b757c56bcef8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:04.462253 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.462232    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb15cdc1-cc17-4c1b-bec5-b757c56bcef8-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc\" (UID: \"fb15cdc1-cc17-4c1b-bec5-b757c56bcef8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:04.462305 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.462269    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb15cdc1-cc17-4c1b-bec5-b757c56bcef8-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc\" (UID: \"fb15cdc1-cc17-4c1b-bec5-b757c56bcef8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:04.462350 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.462317    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb15cdc1-cc17-4c1b-bec5-b757c56bcef8-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc\" (UID: \"fb15cdc1-cc17-4c1b-bec5-b757c56bcef8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:04.464229 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.464202    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb15cdc1-cc17-4c1b-bec5-b757c56bcef8-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc\" (UID: \"fb15cdc1-cc17-4c1b-bec5-b757c56bcef8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:04.464478 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.464459    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb15cdc1-cc17-4c1b-bec5-b757c56bcef8-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc\" (UID: \"fb15cdc1-cc17-4c1b-bec5-b757c56bcef8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:04.478474 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.478447    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw2bd\" (UniqueName: \"kubernetes.io/projected/fb15cdc1-cc17-4c1b-bec5-b757c56bcef8-kube-api-access-hw2bd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc\" (UID: \"fb15cdc1-cc17-4c1b-bec5-b757c56bcef8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:04.585695 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.585611    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:04.877498 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.877471    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"]
Sep 11 16:13:04.880184 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:13:04.880156    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb15cdc1_cc17_4c1b_bec5_b757c56bcef8.slice/crio-2fd3e0c2c355b596d55f1b0e2de0cd7b04e5a7ac739f94f91acd7360710bf403 WatchSource:0}: Error finding container 2fd3e0c2c355b596d55f1b0e2de0cd7b04e5a7ac739f94f91acd7360710bf403: Status 404 returned error can't find the container with id 2fd3e0c2c355b596d55f1b0e2de0cd7b04e5a7ac739f94f91acd7360710bf403
Sep 11 16:13:04.881525 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:04.881505    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:13:05.409630 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:05.409592    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc" event={"ID":"fb15cdc1-cc17-4c1b-bec5-b757c56bcef8","Type":"ContainerStarted","Data":"2fd3e0c2c355b596d55f1b0e2de0cd7b04e5a7ac739f94f91acd7360710bf403"}
Sep 11 16:13:06.416363 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:06.416323    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc" event={"ID":"fb15cdc1-cc17-4c1b-bec5-b757c56bcef8","Type":"ContainerStarted","Data":"2a07ae631a871e7a409ea9a407b6702634ac0b8956744cc7d392501db23999f6"}
Sep 11 16:13:14.586093 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:14.586006    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:14.586602 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:14.586119    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:14.598631 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:14.598594    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:14.631081 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:14.631008    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc" podStartSLOduration=9.625091317999999 podStartE2EDuration="10.630985492s" totalImagesPullingTime="1.005894174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:13:06.455338471 +0000 UTC m=+646.300982026" watchObservedRunningTime="2026-09-11 16:13:14.630985492 +0000 UTC m=+654.476629047"
Sep 11 16:13:15.468516 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:15.468480    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc"
Sep 11 16:13:24.462911 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.462876    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"]
Sep 11 16:13:24.465324 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.465305    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:24.483352 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.483325    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 11 16:13:24.493530 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.493500    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"]
Sep 11 16:13:24.526970 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.526930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8bac3d7-6c0a-4e54-a454-fc78bf587340-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4nzq2\" (UID: \"a8bac3d7-6c0a-4e54-a454-fc78bf587340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:24.527195 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.526988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8bac3d7-6c0a-4e54-a454-fc78bf587340-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4nzq2\" (UID: \"a8bac3d7-6c0a-4e54-a454-fc78bf587340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:24.527195 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.527063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8bac3d7-6c0a-4e54-a454-fc78bf587340-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4nzq2\" (UID: \"a8bac3d7-6c0a-4e54-a454-fc78bf587340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:24.527195 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.527118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bac3d7-6c0a-4e54-a454-fc78bf587340-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4nzq2\" (UID: \"a8bac3d7-6c0a-4e54-a454-fc78bf587340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:24.527195 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.527173    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8bac3d7-6c0a-4e54-a454-fc78bf587340-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4nzq2\" (UID: \"a8bac3d7-6c0a-4e54-a454-fc78bf587340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:24.527367 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.527215    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfqrm\" (UniqueName: \"kubernetes.io/projected/a8bac3d7-6c0a-4e54-a454-fc78bf587340-kube-api-access-vfqrm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4nzq2\" (UID: \"a8bac3d7-6c0a-4e54-a454-fc78bf587340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:24.627698 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.627659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8bac3d7-6c0a-4e54-a454-fc78bf587340-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4nzq2\" (UID: \"a8bac3d7-6c0a-4e54-a454-fc78bf587340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:24.627887 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.627717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8bac3d7-6c0a-4e54-a454-fc78bf587340-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4nzq2\" (UID: \"a8bac3d7-6c0a-4e54-a454-fc78bf587340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:24.627887 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.627742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8bac3d7-6c0a-4e54-a454-fc78bf587340-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4nzq2\" (UID: \"a8bac3d7-6c0a-4e54-a454-fc78bf587340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:24.627887 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.627791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bac3d7-6c0a-4e54-a454-fc78bf587340-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4nzq2\" (UID: \"a8bac3d7-6c0a-4e54-a454-fc78bf587340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:24.627887 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.627839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8bac3d7-6c0a-4e54-a454-fc78bf587340-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4nzq2\" (UID: \"a8bac3d7-6c0a-4e54-a454-fc78bf587340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:24.627887 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.627870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfqrm\" (UniqueName: \"kubernetes.io/projected/a8bac3d7-6c0a-4e54-a454-fc78bf587340-kube-api-access-vfqrm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4nzq2\" (UID: \"a8bac3d7-6c0a-4e54-a454-fc78bf587340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:24.628189 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.628164    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8bac3d7-6c0a-4e54-a454-fc78bf587340-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4nzq2\" (UID: \"a8bac3d7-6c0a-4e54-a454-fc78bf587340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:24.628246 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.628173    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8bac3d7-6c0a-4e54-a454-fc78bf587340-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4nzq2\" (UID: \"a8bac3d7-6c0a-4e54-a454-fc78bf587340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:24.628355 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.628333    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8bac3d7-6c0a-4e54-a454-fc78bf587340-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4nzq2\" (UID: \"a8bac3d7-6c0a-4e54-a454-fc78bf587340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:24.630337 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.630314    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8bac3d7-6c0a-4e54-a454-fc78bf587340-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4nzq2\" (UID: \"a8bac3d7-6c0a-4e54-a454-fc78bf587340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:24.630880 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.630849    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bac3d7-6c0a-4e54-a454-fc78bf587340-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4nzq2\" (UID: \"a8bac3d7-6c0a-4e54-a454-fc78bf587340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:24.636884 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.636857    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfqrm\" (UniqueName: \"kubernetes.io/projected/a8bac3d7-6c0a-4e54-a454-fc78bf587340-kube-api-access-vfqrm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4nzq2\" (UID: \"a8bac3d7-6c0a-4e54-a454-fc78bf587340\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:24.775083 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:24.774938    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:25.051296 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:25.051266    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"]
Sep 11 16:13:25.053529 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:13:25.053502    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bac3d7_6c0a_4e54_a454_fc78bf587340.slice/crio-b97aa048822887ef767352c7067fa248b095384d365e649bb31a971dc258b2c1 WatchSource:0}: Error finding container b97aa048822887ef767352c7067fa248b095384d365e649bb31a971dc258b2c1: Status 404 returned error can't find the container with id b97aa048822887ef767352c7067fa248b095384d365e649bb31a971dc258b2c1
Sep 11 16:13:25.503623 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:25.503585    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2" event={"ID":"a8bac3d7-6c0a-4e54-a454-fc78bf587340","Type":"ContainerStarted","Data":"dc80ce01d032e7962fd8f68e24ae56087ab5df8fdcdc78f7671e16722a8c5c9a"}
Sep 11 16:13:25.504091 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:25.503639    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2" event={"ID":"a8bac3d7-6c0a-4e54-a454-fc78bf587340","Type":"ContainerStarted","Data":"b97aa048822887ef767352c7067fa248b095384d365e649bb31a971dc258b2c1"}
Sep 11 16:13:25.524299 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:25.524239    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2" podStartSLOduration=1.337193213 podStartE2EDuration="1.524216755s" totalImagesPullingTime="187.023542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:13:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:13:25.523664386 +0000 UTC m=+665.369307942" watchObservedRunningTime="2026-09-11 16:13:25.524216755 +0000 UTC m=+665.369860313"
Sep 11 16:13:34.775120 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:34.775078    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:34.775651 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:34.775166    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:34.787980 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:34.787946    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:35.564702 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:35.564666    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4nzq2"
Sep 11 16:13:37.762952 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.762915    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"]
Sep 11 16:13:37.766645 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.766621    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:37.769345 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.769323    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 11 16:13:37.784280 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.784246    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"]
Sep 11 16:13:37.832721 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.832689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jfp6b\" (UID: \"9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:37.832905 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.832747    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jfp6b\" (UID: \"9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:37.832905 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.832812    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jfp6b\" (UID: \"9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:37.832905 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.832844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jfp6b\" (UID: \"9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:37.832905 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.832876    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzkvh\" (UniqueName: \"kubernetes.io/projected/9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b-kube-api-access-nzkvh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jfp6b\" (UID: \"9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:37.833097 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.832914    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jfp6b\" (UID: \"9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:37.933971 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.933936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jfp6b\" (UID: \"9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:37.933971 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.933975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jfp6b\" (UID: \"9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:37.934260 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.933997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkvh\" (UniqueName: \"kubernetes.io/projected/9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b-kube-api-access-nzkvh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jfp6b\" (UID: \"9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:37.934260 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.934019    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jfp6b\" (UID: \"9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:37.934260 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.934088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jfp6b\" (UID: \"9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:37.934260 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.934131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jfp6b\" (UID: \"9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:37.934478 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.934444    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jfp6b\" (UID: \"9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:37.934520 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.934490    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jfp6b\" (UID: \"9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:37.934590 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.934572    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jfp6b\" (UID: \"9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:37.936603 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.936577    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jfp6b\" (UID: \"9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:37.936715 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.936627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jfp6b\" (UID: \"9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:37.942087 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:37.942066    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzkvh\" (UniqueName: \"kubernetes.io/projected/9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b-kube-api-access-nzkvh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jfp6b\" (UID: \"9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:38.081096 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:38.080990    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:38.362947 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:38.362922    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"]
Sep 11 16:13:38.364964 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:13:38.364939    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b97fcf2_48f1_4eb7_9bc1_7b71fb23f61b.slice/crio-962368589c13fee9da886881ecc5cf1fb56b27c09b840e820bd313ec147f8928 WatchSource:0}: Error finding container 962368589c13fee9da886881ecc5cf1fb56b27c09b840e820bd313ec147f8928: Status 404 returned error can't find the container with id 962368589c13fee9da886881ecc5cf1fb56b27c09b840e820bd313ec147f8928
Sep 11 16:13:38.567929 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:38.567898    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b" event={"ID":"9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b","Type":"ContainerStarted","Data":"962368589c13fee9da886881ecc5cf1fb56b27c09b840e820bd313ec147f8928"}
Sep 11 16:13:39.574711 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:39.574675    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b" event={"ID":"9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b","Type":"ContainerStarted","Data":"c4849d73608db89212e2da3e24f94c9725e9514fe383d7ca36dc10310251f7e1"}
Sep 11 16:13:39.596622 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:39.596558    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b" podStartSLOduration=2.404146108 podStartE2EDuration="2.596538376s" totalImagesPullingTime="192.392268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:13:39.59453181 +0000 UTC m=+679.440175378" watchObservedRunningTime="2026-09-11 16:13:39.596538376 +0000 UTC m=+679.442181931"
Sep 11 16:13:48.081206 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:48.081172    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:48.081582 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:48.081237    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:48.093771 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:48.093740    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:13:48.632106 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:13:48.632078    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jfp6b"
Sep 11 16:15:09.536190 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:09.536155    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55fd49fbf5-hctcb"]
Sep 11 16:15:09.539870 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:09.539849    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55fd49fbf5-hctcb"
Sep 11 16:15:09.550991 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:09.550959    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55fd49fbf5-hctcb"]
Sep 11 16:15:09.632813 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:09.632777    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlnm6\" (UniqueName: \"kubernetes.io/projected/e2256359-778b-44b4-8956-62a5a4626466-kube-api-access-zlnm6\") pod \"authorino-55fd49fbf5-hctcb\" (UID: \"e2256359-778b-44b4-8956-62a5a4626466\") " pod="kuadrant-system/authorino-55fd49fbf5-hctcb"
Sep 11 16:15:09.632979 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:09.632821    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2256359-778b-44b4-8956-62a5a4626466-tls-cert\") pod \"authorino-55fd49fbf5-hctcb\" (UID: \"e2256359-778b-44b4-8956-62a5a4626466\") " pod="kuadrant-system/authorino-55fd49fbf5-hctcb"
Sep 11 16:15:09.733757 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:09.733720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlnm6\" (UniqueName: \"kubernetes.io/projected/e2256359-778b-44b4-8956-62a5a4626466-kube-api-access-zlnm6\") pod \"authorino-55fd49fbf5-hctcb\" (UID: \"e2256359-778b-44b4-8956-62a5a4626466\") " pod="kuadrant-system/authorino-55fd49fbf5-hctcb"
Sep 11 16:15:09.733956 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:09.733767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2256359-778b-44b4-8956-62a5a4626466-tls-cert\") pod \"authorino-55fd49fbf5-hctcb\" (UID: \"e2256359-778b-44b4-8956-62a5a4626466\") " pod="kuadrant-system/authorino-55fd49fbf5-hctcb"
Sep 11 16:15:09.736380 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:09.736359    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2256359-778b-44b4-8956-62a5a4626466-tls-cert\") pod \"authorino-55fd49fbf5-hctcb\" (UID: \"e2256359-778b-44b4-8956-62a5a4626466\") " pod="kuadrant-system/authorino-55fd49fbf5-hctcb"
Sep 11 16:15:09.744878 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:09.744854    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlnm6\" (UniqueName: \"kubernetes.io/projected/e2256359-778b-44b4-8956-62a5a4626466-kube-api-access-zlnm6\") pod \"authorino-55fd49fbf5-hctcb\" (UID: \"e2256359-778b-44b4-8956-62a5a4626466\") " pod="kuadrant-system/authorino-55fd49fbf5-hctcb"
Sep 11 16:15:09.849085 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:09.849055    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55fd49fbf5-hctcb"
Sep 11 16:15:10.133293 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:10.133117    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55fd49fbf5-hctcb"]
Sep 11 16:15:10.135955 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:15:10.135928    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2256359_778b_44b4_8956_62a5a4626466.slice/crio-595141f23597e062c02abbe780ec1962bcecd6ef79c8f4131edd2b00154fb4e2 WatchSource:0}: Error finding container 595141f23597e062c02abbe780ec1962bcecd6ef79c8f4131edd2b00154fb4e2: Status 404 returned error can't find the container with id 595141f23597e062c02abbe780ec1962bcecd6ef79c8f4131edd2b00154fb4e2
Sep 11 16:15:11.013011 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:11.012975    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55fd49fbf5-hctcb" event={"ID":"e2256359-778b-44b4-8956-62a5a4626466","Type":"ContainerStarted","Data":"595141f23597e062c02abbe780ec1962bcecd6ef79c8f4131edd2b00154fb4e2"}
Sep 11 16:15:16.040018 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:16.039973    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55fd49fbf5-hctcb" event={"ID":"e2256359-778b-44b4-8956-62a5a4626466","Type":"ContainerStarted","Data":"e7ad41882a1beebebdcf881d9c6f2b44503fc89ee073917c36394135cdd9548c"}
Sep 11 16:15:16.065164 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:16.065089    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-55fd49fbf5-hctcb" podStartSLOduration=1.619421504 podStartE2EDuration="7.065062965s" totalImagesPullingTime="5.445641461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:16.063407004 +0000 UTC m=+775.909050560" watchObservedRunningTime="2026-09-11 16:15:16.065062965 +0000 UTC m=+775.910706524"
Sep 11 16:15:16.107758 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:16.107716    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c75d4b78-jpq7m"]
Sep 11 16:15:16.107946 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:16.107914    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c75d4b78-jpq7m" podUID="e33120ab-3871-462f-af72-18f57df9d160" containerName="authorino" containerID="cri-o://a3e5ec3ed61be393ffc5f6e12967b100c1022669fa2584a9459c6385731a8dec" gracePeriod=30
Sep 11 16:15:16.359091 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:16.359062    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c75d4b78-jpq7m"
Sep 11 16:15:16.494068 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:16.494006    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e33120ab-3871-462f-af72-18f57df9d160-tls-cert\") pod \"e33120ab-3871-462f-af72-18f57df9d160\" (UID: \"e33120ab-3871-462f-af72-18f57df9d160\") "
Sep 11 16:15:16.494251 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:16.494125    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w4j2\" (UniqueName: \"kubernetes.io/projected/e33120ab-3871-462f-af72-18f57df9d160-kube-api-access-4w4j2\") pod \"e33120ab-3871-462f-af72-18f57df9d160\" (UID: \"e33120ab-3871-462f-af72-18f57df9d160\") "
Sep 11 16:15:16.499057 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:16.497483    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33120ab-3871-462f-af72-18f57df9d160-kube-api-access-4w4j2" (OuterVolumeSpecName: "kube-api-access-4w4j2") pod "e33120ab-3871-462f-af72-18f57df9d160" (UID: "e33120ab-3871-462f-af72-18f57df9d160"). InnerVolumeSpecName "kube-api-access-4w4j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:15:16.509777 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:16.509709    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e33120ab-3871-462f-af72-18f57df9d160-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e33120ab-3871-462f-af72-18f57df9d160" (UID: "e33120ab-3871-462f-af72-18f57df9d160"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:15:16.595159 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:16.595096    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w4j2\" (UniqueName: \"kubernetes.io/projected/e33120ab-3871-462f-af72-18f57df9d160-kube-api-access-4w4j2\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:15:16.595159 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:16.595123    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e33120ab-3871-462f-af72-18f57df9d160-tls-cert\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:15:17.046927 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:17.046890    2577 generic.go:356] "Generic (PLEG): container finished" podID="e33120ab-3871-462f-af72-18f57df9d160" containerID="a3e5ec3ed61be393ffc5f6e12967b100c1022669fa2584a9459c6385731a8dec" exitCode=0
Sep 11 16:15:17.046927 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:17.046931    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c75d4b78-jpq7m"
Sep 11 16:15:17.047458 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:17.046976    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c75d4b78-jpq7m" event={"ID":"e33120ab-3871-462f-af72-18f57df9d160","Type":"ContainerDied","Data":"a3e5ec3ed61be393ffc5f6e12967b100c1022669fa2584a9459c6385731a8dec"}
Sep 11 16:15:17.047458 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:17.047017    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c75d4b78-jpq7m" event={"ID":"e33120ab-3871-462f-af72-18f57df9d160","Type":"ContainerDied","Data":"9a9a27eed0945a638c8d4b87783623d326d40e4c1c88e718f183a73bda4d8a37"}
Sep 11 16:15:17.047458 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:17.047059    2577 scope.go:117] "RemoveContainer" containerID="a3e5ec3ed61be393ffc5f6e12967b100c1022669fa2584a9459c6385731a8dec"
Sep 11 16:15:17.056058 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:17.056009    2577 scope.go:117] "RemoveContainer" containerID="a3e5ec3ed61be393ffc5f6e12967b100c1022669fa2584a9459c6385731a8dec"
Sep 11 16:15:17.056406 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:15:17.056382    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e5ec3ed61be393ffc5f6e12967b100c1022669fa2584a9459c6385731a8dec\": container with ID starting with a3e5ec3ed61be393ffc5f6e12967b100c1022669fa2584a9459c6385731a8dec not found: ID does not exist" containerID="a3e5ec3ed61be393ffc5f6e12967b100c1022669fa2584a9459c6385731a8dec"
Sep 11 16:15:17.056494 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:17.056418    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e5ec3ed61be393ffc5f6e12967b100c1022669fa2584a9459c6385731a8dec"} err="failed to get container status \"a3e5ec3ed61be393ffc5f6e12967b100c1022669fa2584a9459c6385731a8dec\": rpc error: code = NotFound desc = could not find container \"a3e5ec3ed61be393ffc5f6e12967b100c1022669fa2584a9459c6385731a8dec\": container with ID starting with a3e5ec3ed61be393ffc5f6e12967b100c1022669fa2584a9459c6385731a8dec not found: ID does not exist"
Sep 11 16:15:17.072615 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:17.072585    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c75d4b78-jpq7m"]
Sep 11 16:15:17.082435 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:17.082406    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c75d4b78-jpq7m"]
Sep 11 16:15:18.632398 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:18.632362    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e33120ab-3871-462f-af72-18f57df9d160" path="/var/lib/kubelet/pods/e33120ab-3871-462f-af72-18f57df9d160/volumes"
Sep 11 16:15:37.938650 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:37.938614    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"]
Sep 11 16:15:37.939271 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:37.938995    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e33120ab-3871-462f-af72-18f57df9d160" containerName="authorino"
Sep 11 16:15:37.939271 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:37.939007    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e33120ab-3871-462f-af72-18f57df9d160" containerName="authorino"
Sep 11 16:15:37.939271 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:37.939083    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e33120ab-3871-462f-af72-18f57df9d160" containerName="authorino"
Sep 11 16:15:37.943522 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:37.943502    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"
Sep 11 16:15:37.948330 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:37.948304    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-99877a-serving-cert\""
Sep 11 16:15:37.956828 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:37.956799    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"]
Sep 11 16:15:37.987649 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:37.987613    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70fa3e6-9860-4a19-9e87-24af505c8e94-maas-api-tls\") pod \"maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss\" (UID: \"f70fa3e6-9860-4a19-9e87-24af505c8e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"
Sep 11 16:15:37.987833 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:37.987669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70fa3e6-9860-4a19-9e87-24af505c8e94-openshift-service-ca\") pod \"maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss\" (UID: \"f70fa3e6-9860-4a19-9e87-24af505c8e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"
Sep 11 16:15:37.987833 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:37.987744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h966\" (UniqueName: \"kubernetes.io/projected/f70fa3e6-9860-4a19-9e87-24af505c8e94-kube-api-access-6h966\") pod \"maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss\" (UID: \"f70fa3e6-9860-4a19-9e87-24af505c8e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"
Sep 11 16:15:37.987833 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:37.987819    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f70fa3e6-9860-4a19-9e87-24af505c8e94-metrics-certs\") pod \"maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss\" (UID: \"f70fa3e6-9860-4a19-9e87-24af505c8e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"
Sep 11 16:15:38.089110 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:38.089071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h966\" (UniqueName: \"kubernetes.io/projected/f70fa3e6-9860-4a19-9e87-24af505c8e94-kube-api-access-6h966\") pod \"maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss\" (UID: \"f70fa3e6-9860-4a19-9e87-24af505c8e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"
Sep 11 16:15:38.089292 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:38.089125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f70fa3e6-9860-4a19-9e87-24af505c8e94-metrics-certs\") pod \"maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss\" (UID: \"f70fa3e6-9860-4a19-9e87-24af505c8e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"
Sep 11 16:15:38.089292 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:38.089183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70fa3e6-9860-4a19-9e87-24af505c8e94-maas-api-tls\") pod \"maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss\" (UID: \"f70fa3e6-9860-4a19-9e87-24af505c8e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"
Sep 11 16:15:38.089292 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:38.089218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70fa3e6-9860-4a19-9e87-24af505c8e94-openshift-service-ca\") pod \"maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss\" (UID: \"f70fa3e6-9860-4a19-9e87-24af505c8e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"
Sep 11 16:15:38.089525 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:15:38.089294    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-serving-cert: secret "maas-api-e2e-inf-a-99877a-serving-cert" not found
Sep 11 16:15:38.089525 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:15:38.089369    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f70fa3e6-9860-4a19-9e87-24af505c8e94-maas-api-tls podName:f70fa3e6-9860-4a19-9e87-24af505c8e94 nodeName:}" failed. No retries permitted until 2026-09-11 16:15:38.589349119 +0000 UTC m=+798.434992658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f70fa3e6-9860-4a19-9e87-24af505c8e94-maas-api-tls") pod "maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss" (UID: "f70fa3e6-9860-4a19-9e87-24af505c8e94") : secret "maas-api-e2e-inf-a-99877a-serving-cert" not found
Sep 11 16:15:38.089899 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:38.089875    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70fa3e6-9860-4a19-9e87-24af505c8e94-openshift-service-ca\") pod \"maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss\" (UID: \"f70fa3e6-9860-4a19-9e87-24af505c8e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"
Sep 11 16:15:38.091855 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:38.091824    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f70fa3e6-9860-4a19-9e87-24af505c8e94-metrics-certs\") pod \"maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss\" (UID: \"f70fa3e6-9860-4a19-9e87-24af505c8e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"
Sep 11 16:15:38.102415 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:38.102384    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h966\" (UniqueName: \"kubernetes.io/projected/f70fa3e6-9860-4a19-9e87-24af505c8e94-kube-api-access-6h966\") pod \"maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss\" (UID: \"f70fa3e6-9860-4a19-9e87-24af505c8e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"
Sep 11 16:15:38.594534 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:38.594493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70fa3e6-9860-4a19-9e87-24af505c8e94-maas-api-tls\") pod \"maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss\" (UID: \"f70fa3e6-9860-4a19-9e87-24af505c8e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"
Sep 11 16:15:38.597030 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:38.596999    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70fa3e6-9860-4a19-9e87-24af505c8e94-maas-api-tls\") pod \"maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss\" (UID: \"f70fa3e6-9860-4a19-9e87-24af505c8e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"
Sep 11 16:15:38.854942 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:38.854911    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"
Sep 11 16:15:39.130963 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:39.130932    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"]
Sep 11 16:15:39.133166 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:15:39.133138    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70fa3e6_9860_4a19_9e87_24af505c8e94.slice/crio-383cd8d85c34755f814a5f3e0525867aab02ebc8129770ad5315424dc6456b43 WatchSource:0}: Error finding container 383cd8d85c34755f814a5f3e0525867aab02ebc8129770ad5315424dc6456b43: Status 404 returned error can't find the container with id 383cd8d85c34755f814a5f3e0525867aab02ebc8129770ad5315424dc6456b43
Sep 11 16:15:39.152393 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:39.152356    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss" event={"ID":"f70fa3e6-9860-4a19-9e87-24af505c8e94","Type":"ContainerStarted","Data":"383cd8d85c34755f814a5f3e0525867aab02ebc8129770ad5315424dc6456b43"}
Sep 11 16:15:40.159387 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:40.159345    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss" event={"ID":"f70fa3e6-9860-4a19-9e87-24af505c8e94","Type":"ContainerStarted","Data":"96936adb087181642bd9f475e5b63b97b5818da936dadab74bf67faa75b8f930"}
Sep 11 16:15:40.159774 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:40.159476    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"
Sep 11 16:15:40.182629 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:40.182576    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss" podStartSLOduration=2.6688788949999998 podStartE2EDuration="3.182556167s" totalImagesPullingTime="513.677272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:40.180765594 +0000 UTC m=+800.026409150" watchObservedRunningTime="2026-09-11 16:15:40.182556167 +0000 UTC m=+800.028199756"
Sep 11 16:15:45.527110 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:45.527018    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"]
Sep 11 16:15:45.530476 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:45.530458    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"
Sep 11 16:15:45.535300 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:45.535275    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-16046061-serving-cert\""
Sep 11 16:15:45.546547 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:45.546508    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"]
Sep 11 16:15:45.657314 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:45.657271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-maas-api-tls\") pod \"maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw\" (UID: \"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"
Sep 11 16:15:45.657314 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:45.657316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-metrics-certs\") pod \"maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw\" (UID: \"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"
Sep 11 16:15:45.657543 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:45.657366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-openshift-service-ca\") pod \"maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw\" (UID: \"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"
Sep 11 16:15:45.657543 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:45.657419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrs66\" (UniqueName: \"kubernetes.io/projected/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-kube-api-access-xrs66\") pod \"maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw\" (UID: \"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"
Sep 11 16:15:45.757807 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:45.757766    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-openshift-service-ca\") pod \"maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw\" (UID: \"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"
Sep 11 16:15:45.758023 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:45.757859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrs66\" (UniqueName: \"kubernetes.io/projected/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-kube-api-access-xrs66\") pod \"maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw\" (UID: \"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"
Sep 11 16:15:45.758023 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:45.757892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-maas-api-tls\") pod \"maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw\" (UID: \"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"
Sep 11 16:15:45.758023 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:45.757928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-metrics-certs\") pod \"maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw\" (UID: \"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"
Sep 11 16:15:45.758507 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:45.758481    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-openshift-service-ca\") pod \"maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw\" (UID: \"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"
Sep 11 16:15:45.760423 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:45.760404    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-metrics-certs\") pod \"maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw\" (UID: \"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"
Sep 11 16:15:45.760517 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:45.760439    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-maas-api-tls\") pod \"maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw\" (UID: \"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"
Sep 11 16:15:45.779238 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:45.779159    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrs66\" (UniqueName: \"kubernetes.io/projected/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-kube-api-access-xrs66\") pod \"maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw\" (UID: \"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"
Sep 11 16:15:45.842227 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:45.842189    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"
Sep 11 16:15:46.134602 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:46.134570    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"]
Sep 11 16:15:46.137142 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:15:46.137103    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7f1aa7_31a0_4a5f_aa42_c5375fc5ca01.slice/crio-e8be1009a7ff549259fb98a021061b5805b0d37173e2e5a9be6f348d8f238343 WatchSource:0}: Error finding container e8be1009a7ff549259fb98a021061b5805b0d37173e2e5a9be6f348d8f238343: Status 404 returned error can't find the container with id e8be1009a7ff549259fb98a021061b5805b0d37173e2e5a9be6f348d8f238343
Sep 11 16:15:46.170225 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:46.170192    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"
Sep 11 16:15:46.187674 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:46.187641    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw" event={"ID":"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01","Type":"ContainerStarted","Data":"e8be1009a7ff549259fb98a021061b5805b0d37173e2e5a9be6f348d8f238343"}
Sep 11 16:15:46.432603 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:46.432517    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq"]
Sep 11 16:15:46.437151 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:46.437124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq"
Sep 11 16:15:46.443012 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:46.442986    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-16046061\""
Sep 11 16:15:46.443233 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:46.443092    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-16046061-dockercfg-k5cwd\""
Sep 11 16:15:46.450659 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:46.450625    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq"]
Sep 11 16:15:46.565162 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:46.565122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mqbs\" (UniqueName: \"kubernetes.io/projected/356eb690-2db2-472b-9bb8-d0b26b916996-kube-api-access-6mqbs\") pod \"payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq\" (UID: \"356eb690-2db2-472b-9bb8-d0b26b916996\") " pod="openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq"
Sep 11 16:15:46.565575 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:46.565270    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356eb690-2db2-472b-9bb8-d0b26b916996-plugins-config-volume\") pod \"payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq\" (UID: \"356eb690-2db2-472b-9bb8-d0b26b916996\") " pod="openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq"
Sep 11 16:15:46.665702 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:46.665666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356eb690-2db2-472b-9bb8-d0b26b916996-plugins-config-volume\") pod \"payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq\" (UID: \"356eb690-2db2-472b-9bb8-d0b26b916996\") " pod="openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq"
Sep 11 16:15:46.665811 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:46.665730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mqbs\" (UniqueName: \"kubernetes.io/projected/356eb690-2db2-472b-9bb8-d0b26b916996-kube-api-access-6mqbs\") pod \"payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq\" (UID: \"356eb690-2db2-472b-9bb8-d0b26b916996\") " pod="openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq"
Sep 11 16:15:46.666415 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:46.666392    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356eb690-2db2-472b-9bb8-d0b26b916996-plugins-config-volume\") pod \"payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq\" (UID: \"356eb690-2db2-472b-9bb8-d0b26b916996\") " pod="openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq"
Sep 11 16:15:46.676358 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:46.676322    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mqbs\" (UniqueName: \"kubernetes.io/projected/356eb690-2db2-472b-9bb8-d0b26b916996-kube-api-access-6mqbs\") pod \"payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq\" (UID: \"356eb690-2db2-472b-9bb8-d0b26b916996\") " pod="openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq"
Sep 11 16:15:46.749707 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:46.749667    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq"
Sep 11 16:15:47.042543 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:47.042515    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq"]
Sep 11 16:15:47.044440 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:15:47.044411    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356eb690_2db2_472b_9bb8_d0b26b916996.slice/crio-e8990c2c7cc11f8e048a0b27746378180a890ce664ef1cc90b38332654c88f3e WatchSource:0}: Error finding container e8990c2c7cc11f8e048a0b27746378180a890ce664ef1cc90b38332654c88f3e: Status 404 returned error can't find the container with id e8990c2c7cc11f8e048a0b27746378180a890ce664ef1cc90b38332654c88f3e
Sep 11 16:15:47.194750 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:47.194704    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw" event={"ID":"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01","Type":"ContainerStarted","Data":"73ef1d54b0e9efe61ece0eaab765148171eeed86546bf1ebda8355201b6c3266"}
Sep 11 16:15:47.194902 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:47.194774    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"
Sep 11 16:15:47.196258 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:47.196231    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq" event={"ID":"356eb690-2db2-472b-9bb8-d0b26b916996","Type":"ContainerStarted","Data":"a5a60d2074c94c328707805066afeca21355340d6342758393265a8d1b417dfc"}
Sep 11 16:15:47.196369 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:47.196281    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq" event={"ID":"356eb690-2db2-472b-9bb8-d0b26b916996","Type":"ContainerStarted","Data":"e8990c2c7cc11f8e048a0b27746378180a890ce664ef1cc90b38332654c88f3e"}
Sep 11 16:15:47.196369 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:47.196330    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq"
Sep 11 16:15:47.241804 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:47.241743    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw" podStartSLOduration=1.728781878 podStartE2EDuration="2.241726457s" totalImagesPullingTime="512.944579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:47.240410268 +0000 UTC m=+807.086053824" watchObservedRunningTime="2026-09-11 16:15:47.241726457 +0000 UTC m=+807.087370051"
Sep 11 16:15:47.290235 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:47.290176    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq" podStartSLOduration=1.290156734 podStartE2EDuration="1.290156734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:47.286853569 +0000 UTC m=+807.132497124" watchObservedRunningTime="2026-09-11 16:15:47.290156734 +0000 UTC m=+807.135800289"
Sep 11 16:15:50.856447 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:50.856412    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"]
Sep 11 16:15:50.860101 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:50.860079    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"
Sep 11 16:15:50.864367 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:50.864346    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-e12be0-serving-cert\""
Sep 11 16:15:50.881905 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:50.881872    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"]
Sep 11 16:15:51.002501 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:51.002455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d08bd47-2863-41a6-a6bf-23725684c3b3-maas-api-tls\") pod \"maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m\" (UID: \"7d08bd47-2863-41a6-a6bf-23725684c3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"
Sep 11 16:15:51.002701 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:51.002518    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghb99\" (UniqueName: \"kubernetes.io/projected/7d08bd47-2863-41a6-a6bf-23725684c3b3-kube-api-access-ghb99\") pod \"maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m\" (UID: \"7d08bd47-2863-41a6-a6bf-23725684c3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"
Sep 11 16:15:51.002701 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:51.002539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d08bd47-2863-41a6-a6bf-23725684c3b3-openshift-service-ca\") pod \"maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m\" (UID: \"7d08bd47-2863-41a6-a6bf-23725684c3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"
Sep 11 16:15:51.002701 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:51.002589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d08bd47-2863-41a6-a6bf-23725684c3b3-metrics-certs\") pod \"maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m\" (UID: \"7d08bd47-2863-41a6-a6bf-23725684c3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"
Sep 11 16:15:51.103974 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:51.103935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghb99\" (UniqueName: \"kubernetes.io/projected/7d08bd47-2863-41a6-a6bf-23725684c3b3-kube-api-access-ghb99\") pod \"maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m\" (UID: \"7d08bd47-2863-41a6-a6bf-23725684c3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"
Sep 11 16:15:51.103974 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:51.103975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d08bd47-2863-41a6-a6bf-23725684c3b3-openshift-service-ca\") pod \"maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m\" (UID: \"7d08bd47-2863-41a6-a6bf-23725684c3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"
Sep 11 16:15:51.104213 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:51.104189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d08bd47-2863-41a6-a6bf-23725684c3b3-metrics-certs\") pod \"maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m\" (UID: \"7d08bd47-2863-41a6-a6bf-23725684c3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"
Sep 11 16:15:51.104319 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:51.104302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d08bd47-2863-41a6-a6bf-23725684c3b3-maas-api-tls\") pod \"maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m\" (UID: \"7d08bd47-2863-41a6-a6bf-23725684c3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"
Sep 11 16:15:51.110105 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:15:51.104986    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-serving-cert: secret "maas-api-e2e-autores-e12be0-serving-cert" not found
Sep 11 16:15:51.110105 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:51.105072    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d08bd47-2863-41a6-a6bf-23725684c3b3-openshift-service-ca\") pod \"maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m\" (UID: \"7d08bd47-2863-41a6-a6bf-23725684c3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"
Sep 11 16:15:51.110105 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:15:51.105114    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d08bd47-2863-41a6-a6bf-23725684c3b3-maas-api-tls podName:7d08bd47-2863-41a6-a6bf-23725684c3b3 nodeName:}" failed. No retries permitted until 2026-09-11 16:15:51.605081084 +0000 UTC m=+811.450724639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d08bd47-2863-41a6-a6bf-23725684c3b3-maas-api-tls") pod "maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m" (UID: "7d08bd47-2863-41a6-a6bf-23725684c3b3") : secret "maas-api-e2e-autores-e12be0-serving-cert" not found
Sep 11 16:15:51.110105 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:51.107777    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d08bd47-2863-41a6-a6bf-23725684c3b3-metrics-certs\") pod \"maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m\" (UID: \"7d08bd47-2863-41a6-a6bf-23725684c3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"
Sep 11 16:15:51.124830 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:51.124798    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghb99\" (UniqueName: \"kubernetes.io/projected/7d08bd47-2863-41a6-a6bf-23725684c3b3-kube-api-access-ghb99\") pod \"maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m\" (UID: \"7d08bd47-2863-41a6-a6bf-23725684c3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"
Sep 11 16:15:51.610175 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:51.610138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d08bd47-2863-41a6-a6bf-23725684c3b3-maas-api-tls\") pod \"maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m\" (UID: \"7d08bd47-2863-41a6-a6bf-23725684c3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"
Sep 11 16:15:51.612974 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:51.612936    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d08bd47-2863-41a6-a6bf-23725684c3b3-maas-api-tls\") pod \"maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m\" (UID: \"7d08bd47-2863-41a6-a6bf-23725684c3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"
Sep 11 16:15:51.770225 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:51.770094    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"
Sep 11 16:15:51.958281 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:51.958199    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj"]
Sep 11 16:15:51.962804 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:51.962782    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj"
Sep 11 16:15:51.965571 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:51.965549    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-e12be0-dockercfg-5bgxr\""
Sep 11 16:15:51.966079 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:51.966028    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-e12be0\""
Sep 11 16:15:51.974743 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:51.974720    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj"]
Sep 11 16:15:52.072392 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:52.072363    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"]
Sep 11 16:15:52.073932 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:15:52.073896    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d08bd47_2863_41a6_a6bf_23725684c3b3.slice/crio-2feaaf2b2c8d4c75a0b1e586a47cfb2898c2a9724bf18d3f4da74dde55e92cab WatchSource:0}: Error finding container 2feaaf2b2c8d4c75a0b1e586a47cfb2898c2a9724bf18d3f4da74dde55e92cab: Status 404 returned error can't find the container with id 2feaaf2b2c8d4c75a0b1e586a47cfb2898c2a9724bf18d3f4da74dde55e92cab
Sep 11 16:15:52.115447 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:52.115404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xflpt\" (UniqueName: \"kubernetes.io/projected/62109c5b-7cf9-4040-b0f3-f7c2e5ba6024-kube-api-access-xflpt\") pod \"payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj\" (UID: \"62109c5b-7cf9-4040-b0f3-f7c2e5ba6024\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj"
Sep 11 16:15:52.115621 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:52.115452    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62109c5b-7cf9-4040-b0f3-f7c2e5ba6024-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj\" (UID: \"62109c5b-7cf9-4040-b0f3-f7c2e5ba6024\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj"
Sep 11 16:15:52.216728 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:52.216633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xflpt\" (UniqueName: \"kubernetes.io/projected/62109c5b-7cf9-4040-b0f3-f7c2e5ba6024-kube-api-access-xflpt\") pod \"payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj\" (UID: \"62109c5b-7cf9-4040-b0f3-f7c2e5ba6024\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj"
Sep 11 16:15:52.216728 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:52.216694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62109c5b-7cf9-4040-b0f3-f7c2e5ba6024-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj\" (UID: \"62109c5b-7cf9-4040-b0f3-f7c2e5ba6024\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj"
Sep 11 16:15:52.217601 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:52.217574    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62109c5b-7cf9-4040-b0f3-f7c2e5ba6024-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj\" (UID: \"62109c5b-7cf9-4040-b0f3-f7c2e5ba6024\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj"
Sep 11 16:15:52.224998 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:52.224966    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m" event={"ID":"7d08bd47-2863-41a6-a6bf-23725684c3b3","Type":"ContainerStarted","Data":"2feaaf2b2c8d4c75a0b1e586a47cfb2898c2a9724bf18d3f4da74dde55e92cab"}
Sep 11 16:15:52.226642 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:52.226621    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xflpt\" (UniqueName: \"kubernetes.io/projected/62109c5b-7cf9-4040-b0f3-f7c2e5ba6024-kube-api-access-xflpt\") pod \"payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj\" (UID: \"62109c5b-7cf9-4040-b0f3-f7c2e5ba6024\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj"
Sep 11 16:15:52.275490 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:52.275449    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj"
Sep 11 16:15:52.566129 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:52.566080    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj"]
Sep 11 16:15:52.568248 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:15:52.568221    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62109c5b_7cf9_4040_b0f3_f7c2e5ba6024.slice/crio-a4650789ab1a8582d305197de0dab9a18e03b9b18c2c06bfe785b0dfa93ac96d WatchSource:0}: Error finding container a4650789ab1a8582d305197de0dab9a18e03b9b18c2c06bfe785b0dfa93ac96d: Status 404 returned error can't find the container with id a4650789ab1a8582d305197de0dab9a18e03b9b18c2c06bfe785b0dfa93ac96d
Sep 11 16:15:53.208135 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:53.208102    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"
Sep 11 16:15:53.234842 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:53.234798    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m" event={"ID":"7d08bd47-2863-41a6-a6bf-23725684c3b3","Type":"ContainerStarted","Data":"7a52246c685538fb18b40e41c53ba410154de1683139599b3df9f2c95f6e504d"}
Sep 11 16:15:53.235058 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:53.234960    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"
Sep 11 16:15:53.236416 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:53.236390    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj" event={"ID":"62109c5b-7cf9-4040-b0f3-f7c2e5ba6024","Type":"ContainerStarted","Data":"1eb5cd39cf78c12062a9d9724a62fbfec44f5762a8160075322e10b4a5eb3d01"}
Sep 11 16:15:53.236534 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:53.236435    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj" event={"ID":"62109c5b-7cf9-4040-b0f3-f7c2e5ba6024","Type":"ContainerStarted","Data":"a4650789ab1a8582d305197de0dab9a18e03b9b18c2c06bfe785b0dfa93ac96d"}
Sep 11 16:15:53.236534 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:53.236475    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj"
Sep 11 16:15:53.263049 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:53.262963    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m" podStartSLOduration=2.936561207 podStartE2EDuration="3.262938442s" totalImagesPullingTime="326.377235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:53.261147857 +0000 UTC m=+813.106791424" watchObservedRunningTime="2026-09-11 16:15:53.262938442 +0000 UTC m=+813.108581999"
Sep 11 16:15:53.290855 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:53.290801    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj" podStartSLOduration=2.290782699 podStartE2EDuration="2.290782699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:53.288507861 +0000 UTC m=+813.134151416" watchObservedRunningTime="2026-09-11 16:15:53.290782699 +0000 UTC m=+813.136426254"
Sep 11 16:15:58.204627 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:58.204596    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq"
Sep 11 16:15:59.248516 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:15:59.248487    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"
Sep 11 16:16:00.325612 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:00.325577    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n"]
Sep 11 16:16:00.330251 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:00.330224    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n"
Sep 11 16:16:00.333451 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:00.333425    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b6b139\""
Sep 11 16:16:00.333572 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:00.333470    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b6b139-dockercfg-kmk5n\""
Sep 11 16:16:00.345879 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:00.345845    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n"]
Sep 11 16:16:00.497429 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:00.497395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hsq8\" (UniqueName: \"kubernetes.io/projected/d6267331-2bd6-479b-957a-656400a77b57-kube-api-access-6hsq8\") pod \"payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n\" (UID: \"d6267331-2bd6-479b-957a-656400a77b57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n"
Sep 11 16:16:00.497429 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:00.497432    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6267331-2bd6-479b-957a-656400a77b57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n\" (UID: \"d6267331-2bd6-479b-957a-656400a77b57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n"
Sep 11 16:16:00.598447 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:00.598415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6267331-2bd6-479b-957a-656400a77b57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n\" (UID: \"d6267331-2bd6-479b-957a-656400a77b57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n"
Sep 11 16:16:00.598636 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:00.598519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hsq8\" (UniqueName: \"kubernetes.io/projected/d6267331-2bd6-479b-957a-656400a77b57-kube-api-access-6hsq8\") pod \"payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n\" (UID: \"d6267331-2bd6-479b-957a-656400a77b57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n"
Sep 11 16:16:00.599206 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:00.599181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6267331-2bd6-479b-957a-656400a77b57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n\" (UID: \"d6267331-2bd6-479b-957a-656400a77b57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n"
Sep 11 16:16:00.613576 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:00.613536    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hsq8\" (UniqueName: \"kubernetes.io/projected/d6267331-2bd6-479b-957a-656400a77b57-kube-api-access-6hsq8\") pod \"payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n\" (UID: \"d6267331-2bd6-479b-957a-656400a77b57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n"
Sep 11 16:16:00.643195 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:00.643161    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n"
Sep 11 16:16:00.938217 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:00.938187    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n"]
Sep 11 16:16:00.940110 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:16:00.940080    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6267331_2bd6_479b_957a_656400a77b57.slice/crio-cfcd6c8614a144d2741250bb9e31e0306e4d2b572d9c7db6dd0159c8446822c0 WatchSource:0}: Error finding container cfcd6c8614a144d2741250bb9e31e0306e4d2b572d9c7db6dd0159c8446822c0: Status 404 returned error can't find the container with id cfcd6c8614a144d2741250bb9e31e0306e4d2b572d9c7db6dd0159c8446822c0
Sep 11 16:16:01.280377 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:01.280273    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n" event={"ID":"d6267331-2bd6-479b-957a-656400a77b57","Type":"ContainerStarted","Data":"6bdf236d3b57adc56817bc96a1ec12d676bb581b7099346260730a3fc57c6a0a"}
Sep 11 16:16:01.280377 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:01.280328    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n" event={"ID":"d6267331-2bd6-479b-957a-656400a77b57","Type":"ContainerStarted","Data":"cfcd6c8614a144d2741250bb9e31e0306e4d2b572d9c7db6dd0159c8446822c0"}
Sep 11 16:16:01.280617 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:01.280398    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n"
Sep 11 16:16:01.314756 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:01.314684    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n" podStartSLOduration=1.3146615050000001 podStartE2EDuration="1.314661505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:01.313345031 +0000 UTC m=+821.158988587" watchObservedRunningTime="2026-09-11 16:16:01.314661505 +0000 UTC m=+821.160305171"
Sep 11 16:16:04.244870 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:04.244838    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj"
Sep 11 16:16:10.661737 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:10.661700    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"]
Sep 11 16:16:10.662265 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:10.661904    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw" podUID="5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01" containerName="maas-api" containerID="cri-o://73ef1d54b0e9efe61ece0eaab765148171eeed86546bf1ebda8355201b6c3266" gracePeriod=30
Sep 11 16:16:10.766555 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:10.766523    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq"]
Sep 11 16:16:10.766812 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:10.766763    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq" podUID="356eb690-2db2-472b-9bb8-d0b26b916996" containerName="payload-processing" containerID="cri-o://a5a60d2074c94c328707805066afeca21355340d6342758393265a8d1b417dfc" gracePeriod=30
Sep 11 16:16:10.950464 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:10.950169    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"
Sep 11 16:16:10.982873 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:10.982230    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-metrics-certs\") pod \"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01\" (UID: \"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01\") "
Sep 11 16:16:10.982873 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:10.982274    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-openshift-service-ca\") pod \"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01\" (UID: \"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01\") "
Sep 11 16:16:10.982873 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:10.982310    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrs66\" (UniqueName: \"kubernetes.io/projected/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-kube-api-access-xrs66\") pod \"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01\" (UID: \"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01\") "
Sep 11 16:16:10.982873 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:10.982360    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-maas-api-tls\") pod \"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01\" (UID: \"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01\") "
Sep 11 16:16:10.982873 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:10.982856    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01" (UID: "5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:16:10.985675 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:10.985644    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01" (UID: "5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:16:10.985915 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:10.985887    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-kube-api-access-xrs66" (OuterVolumeSpecName: "kube-api-access-xrs66") pod "5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01" (UID: "5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01"). InnerVolumeSpecName "kube-api-access-xrs66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:16:10.986344 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:10.986313    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01" (UID: "5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:16:11.029375 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.029347    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq"
Sep 11 16:16:11.083321 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.083287    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mqbs\" (UniqueName: \"kubernetes.io/projected/356eb690-2db2-472b-9bb8-d0b26b916996-kube-api-access-6mqbs\") pod \"356eb690-2db2-472b-9bb8-d0b26b916996\" (UID: \"356eb690-2db2-472b-9bb8-d0b26b916996\") "
Sep 11 16:16:11.083490 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.083403    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356eb690-2db2-472b-9bb8-d0b26b916996-plugins-config-volume\") pod \"356eb690-2db2-472b-9bb8-d0b26b916996\" (UID: \"356eb690-2db2-472b-9bb8-d0b26b916996\") "
Sep 11 16:16:11.083574 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.083563    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-metrics-certs\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:16:11.083609 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.083576    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-openshift-service-ca\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:16:11.083609 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.083587    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrs66\" (UniqueName: \"kubernetes.io/projected/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-kube-api-access-xrs66\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:16:11.083609 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.083595    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01-maas-api-tls\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:16:11.083798 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.083769    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/356eb690-2db2-472b-9bb8-d0b26b916996-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "356eb690-2db2-472b-9bb8-d0b26b916996" (UID: "356eb690-2db2-472b-9bb8-d0b26b916996"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:16:11.085669 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.085639    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/356eb690-2db2-472b-9bb8-d0b26b916996-kube-api-access-6mqbs" (OuterVolumeSpecName: "kube-api-access-6mqbs") pod "356eb690-2db2-472b-9bb8-d0b26b916996" (UID: "356eb690-2db2-472b-9bb8-d0b26b916996"). InnerVolumeSpecName "kube-api-access-6mqbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:16:11.184283 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.184189    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356eb690-2db2-472b-9bb8-d0b26b916996-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:16:11.184283 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.184225    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mqbs\" (UniqueName: \"kubernetes.io/projected/356eb690-2db2-472b-9bb8-d0b26b916996-kube-api-access-6mqbs\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:16:11.333798 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.333765    2577 generic.go:356] "Generic (PLEG): container finished" podID="5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01" containerID="73ef1d54b0e9efe61ece0eaab765148171eeed86546bf1ebda8355201b6c3266" exitCode=0
Sep 11 16:16:11.333984 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.333827    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"
Sep 11 16:16:11.333984 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.333847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw" event={"ID":"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01","Type":"ContainerDied","Data":"73ef1d54b0e9efe61ece0eaab765148171eeed86546bf1ebda8355201b6c3266"}
Sep 11 16:16:11.333984 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.333951    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw" event={"ID":"5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01","Type":"ContainerDied","Data":"e8be1009a7ff549259fb98a021061b5805b0d37173e2e5a9be6f348d8f238343"}
Sep 11 16:16:11.334171 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.333985    2577 scope.go:117] "RemoveContainer" containerID="73ef1d54b0e9efe61ece0eaab765148171eeed86546bf1ebda8355201b6c3266"
Sep 11 16:16:11.335521 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.335498    2577 generic.go:356] "Generic (PLEG): container finished" podID="356eb690-2db2-472b-9bb8-d0b26b916996" containerID="a5a60d2074c94c328707805066afeca21355340d6342758393265a8d1b417dfc" exitCode=0
Sep 11 16:16:11.335642 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.335562    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq"
Sep 11 16:16:11.335701 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.335577    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq" event={"ID":"356eb690-2db2-472b-9bb8-d0b26b916996","Type":"ContainerDied","Data":"a5a60d2074c94c328707805066afeca21355340d6342758393265a8d1b417dfc"}
Sep 11 16:16:11.335753 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.335699    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq" event={"ID":"356eb690-2db2-472b-9bb8-d0b26b916996","Type":"ContainerDied","Data":"e8990c2c7cc11f8e048a0b27746378180a890ce664ef1cc90b38332654c88f3e"}
Sep 11 16:16:11.345848 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.345824    2577 scope.go:117] "RemoveContainer" containerID="73ef1d54b0e9efe61ece0eaab765148171eeed86546bf1ebda8355201b6c3266"
Sep 11 16:16:11.346227 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:16:11.346192    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ef1d54b0e9efe61ece0eaab765148171eeed86546bf1ebda8355201b6c3266\": container with ID starting with 73ef1d54b0e9efe61ece0eaab765148171eeed86546bf1ebda8355201b6c3266 not found: ID does not exist" containerID="73ef1d54b0e9efe61ece0eaab765148171eeed86546bf1ebda8355201b6c3266"
Sep 11 16:16:11.346334 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.346231    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ef1d54b0e9efe61ece0eaab765148171eeed86546bf1ebda8355201b6c3266"} err="failed to get container status \"73ef1d54b0e9efe61ece0eaab765148171eeed86546bf1ebda8355201b6c3266\": rpc error: code = NotFound desc = could not find container \"73ef1d54b0e9efe61ece0eaab765148171eeed86546bf1ebda8355201b6c3266\": container with ID starting with 73ef1d54b0e9efe61ece0eaab765148171eeed86546bf1ebda8355201b6c3266 not found: ID does not exist"
Sep 11 16:16:11.346334 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.346258    2577 scope.go:117] "RemoveContainer" containerID="a5a60d2074c94c328707805066afeca21355340d6342758393265a8d1b417dfc"
Sep 11 16:16:11.354472 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.354434    2577 scope.go:117] "RemoveContainer" containerID="a5a60d2074c94c328707805066afeca21355340d6342758393265a8d1b417dfc"
Sep 11 16:16:11.354712 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:16:11.354693    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a60d2074c94c328707805066afeca21355340d6342758393265a8d1b417dfc\": container with ID starting with a5a60d2074c94c328707805066afeca21355340d6342758393265a8d1b417dfc not found: ID does not exist" containerID="a5a60d2074c94c328707805066afeca21355340d6342758393265a8d1b417dfc"
Sep 11 16:16:11.354769 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.354717    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a60d2074c94c328707805066afeca21355340d6342758393265a8d1b417dfc"} err="failed to get container status \"a5a60d2074c94c328707805066afeca21355340d6342758393265a8d1b417dfc\": rpc error: code = NotFound desc = could not find container \"a5a60d2074c94c328707805066afeca21355340d6342758393265a8d1b417dfc\": container with ID starting with a5a60d2074c94c328707805066afeca21355340d6342758393265a8d1b417dfc not found: ID does not exist"
Sep 11 16:16:11.377740 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.377710    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq"]
Sep 11 16:16:11.387854 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.387828    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-16046061-5b5b4dfbff-nswbq"]
Sep 11 16:16:11.413049 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.412998    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"]
Sep 11 16:16:11.416909 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:11.416884    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16046061-c4d8bfbdb-p6fjw"]
Sep 11 16:16:12.288418 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:12.288392    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n"
Sep 11 16:16:12.633106 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:12.633078    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="356eb690-2db2-472b-9bb8-d0b26b916996" path="/var/lib/kubelet/pods/356eb690-2db2-472b-9bb8-d0b26b916996/volumes"
Sep 11 16:16:12.633474 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:12.633461    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01" path="/var/lib/kubelet/pods/5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01/volumes"
Sep 11 16:16:21.837947 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.837902    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t"]
Sep 11 16:16:21.838342 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.838321    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01" containerName="maas-api"
Sep 11 16:16:21.838342 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.838334    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01" containerName="maas-api"
Sep 11 16:16:21.838418 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.838352    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="356eb690-2db2-472b-9bb8-d0b26b916996" containerName="payload-processing"
Sep 11 16:16:21.838418 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.838358    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="356eb690-2db2-472b-9bb8-d0b26b916996" containerName="payload-processing"
Sep 11 16:16:21.838418 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.838413    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="356eb690-2db2-472b-9bb8-d0b26b916996" containerName="payload-processing"
Sep 11 16:16:21.838511 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.838420    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f7f1aa7-31a0-4a5f-aa42-c5375fc5ca01" containerName="maas-api"
Sep 11 16:16:21.842763 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.842744    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t"
Sep 11 16:16:21.851756 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.851735    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-02a2f652-dockercfg-jjzw2\""
Sep 11 16:16:21.852580 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.852561    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-02a2f652\""
Sep 11 16:16:21.863535 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.863505    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t"]
Sep 11 16:16:21.882598 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.882564    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb4f5654-5483-4afb-92b9-38c388417a86-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t\" (UID: \"eb4f5654-5483-4afb-92b9-38c388417a86\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t"
Sep 11 16:16:21.882764 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.882639    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrnc2\" (UniqueName: \"kubernetes.io/projected/eb4f5654-5483-4afb-92b9-38c388417a86-kube-api-access-qrnc2\") pod \"payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t\" (UID: \"eb4f5654-5483-4afb-92b9-38c388417a86\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t"
Sep 11 16:16:21.910894 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.910857    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk"]
Sep 11 16:16:21.914940 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.914918    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk"
Sep 11 16:16:21.930732 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.930702    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk"]
Sep 11 16:16:21.983325 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.983283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrnc2\" (UniqueName: \"kubernetes.io/projected/eb4f5654-5483-4afb-92b9-38c388417a86-kube-api-access-qrnc2\") pod \"payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t\" (UID: \"eb4f5654-5483-4afb-92b9-38c388417a86\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t"
Sep 11 16:16:21.983528 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.983356    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f67beb5-0a9d-497f-8c64-a9b114757652-plugins-config-volume\") pod \"payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk\" (UID: \"5f67beb5-0a9d-497f-8c64-a9b114757652\") " pod="openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk"
Sep 11 16:16:21.983528 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.983378    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxdjl\" (UniqueName: \"kubernetes.io/projected/5f67beb5-0a9d-497f-8c64-a9b114757652-kube-api-access-hxdjl\") pod \"payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk\" (UID: \"5f67beb5-0a9d-497f-8c64-a9b114757652\") " pod="openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk"
Sep 11 16:16:21.983528 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.983397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb4f5654-5483-4afb-92b9-38c388417a86-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t\" (UID: \"eb4f5654-5483-4afb-92b9-38c388417a86\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t"
Sep 11 16:16:21.984133 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.984113    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb4f5654-5483-4afb-92b9-38c388417a86-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t\" (UID: \"eb4f5654-5483-4afb-92b9-38c388417a86\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t"
Sep 11 16:16:21.996411 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:21.996381    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrnc2\" (UniqueName: \"kubernetes.io/projected/eb4f5654-5483-4afb-92b9-38c388417a86-kube-api-access-qrnc2\") pod \"payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t\" (UID: \"eb4f5654-5483-4afb-92b9-38c388417a86\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t"
Sep 11 16:16:22.084153 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:22.084109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f67beb5-0a9d-497f-8c64-a9b114757652-plugins-config-volume\") pod \"payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk\" (UID: \"5f67beb5-0a9d-497f-8c64-a9b114757652\") " pod="openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk"
Sep 11 16:16:22.084153 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:22.084155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxdjl\" (UniqueName: \"kubernetes.io/projected/5f67beb5-0a9d-497f-8c64-a9b114757652-kube-api-access-hxdjl\") pod \"payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk\" (UID: \"5f67beb5-0a9d-497f-8c64-a9b114757652\") " pod="openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk"
Sep 11 16:16:22.084808 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:22.084784    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f67beb5-0a9d-497f-8c64-a9b114757652-plugins-config-volume\") pod \"payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk\" (UID: \"5f67beb5-0a9d-497f-8c64-a9b114757652\") " pod="openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk"
Sep 11 16:16:22.093541 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:22.093477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxdjl\" (UniqueName: \"kubernetes.io/projected/5f67beb5-0a9d-497f-8c64-a9b114757652-kube-api-access-hxdjl\") pod \"payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk\" (UID: \"5f67beb5-0a9d-497f-8c64-a9b114757652\") " pod="openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk"
Sep 11 16:16:22.152873 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:22.152833    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t"
Sep 11 16:16:22.227018 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:22.226863    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk"
Sep 11 16:16:22.435020 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:22.434996    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t"]
Sep 11 16:16:22.436296 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:16:22.436270    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4f5654_5483_4afb_92b9_38c388417a86.slice/crio-54dfedbce43bcfe19005ae76314e4c5d8bb02047c19ed60d46566219dba10e41 WatchSource:0}: Error finding container 54dfedbce43bcfe19005ae76314e4c5d8bb02047c19ed60d46566219dba10e41: Status 404 returned error can't find the container with id 54dfedbce43bcfe19005ae76314e4c5d8bb02047c19ed60d46566219dba10e41
Sep 11 16:16:22.516941 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:22.516907    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk"]
Sep 11 16:16:22.521528 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:16:22.521494    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f67beb5_0a9d_497f_8c64_a9b114757652.slice/crio-f67ff1d4b82589107a76556fb7494825e0d9f6fd16fa8ad9636a1dbefdca6cd6 WatchSource:0}: Error finding container f67ff1d4b82589107a76556fb7494825e0d9f6fd16fa8ad9636a1dbefdca6cd6: Status 404 returned error can't find the container with id f67ff1d4b82589107a76556fb7494825e0d9f6fd16fa8ad9636a1dbefdca6cd6
Sep 11 16:16:23.396944 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:23.396906    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk" event={"ID":"5f67beb5-0a9d-497f-8c64-a9b114757652","Type":"ContainerStarted","Data":"6795952cfc5d61b01a37d7ae9aa17323e8ad3d495addf102f91d7265820696ce"}
Sep 11 16:16:23.397454 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:23.397136    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk"
Sep 11 16:16:23.397454 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:23.397161    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk" event={"ID":"5f67beb5-0a9d-497f-8c64-a9b114757652","Type":"ContainerStarted","Data":"f67ff1d4b82589107a76556fb7494825e0d9f6fd16fa8ad9636a1dbefdca6cd6"}
Sep 11 16:16:23.398543 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:23.398519    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t" event={"ID":"eb4f5654-5483-4afb-92b9-38c388417a86","Type":"ContainerStarted","Data":"3235d4a842cddf3cec67ea5407c5059a5ee110e640a2969b4d5f9d531fa14072"}
Sep 11 16:16:23.398654 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:23.398557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t" event={"ID":"eb4f5654-5483-4afb-92b9-38c388417a86","Type":"ContainerStarted","Data":"54dfedbce43bcfe19005ae76314e4c5d8bb02047c19ed60d46566219dba10e41"}
Sep 11 16:16:23.398654 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:23.398595    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t"
Sep 11 16:16:23.418474 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:23.418423    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk" podStartSLOduration=2.41840668 podStartE2EDuration="2.41840668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:23.415881065 +0000 UTC m=+843.261524620" watchObservedRunningTime="2026-09-11 16:16:23.41840668 +0000 UTC m=+843.264050235"
Sep 11 16:16:34.406221 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:34.406182    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t"
Sep 11 16:16:34.406800 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:34.406261    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk"
Sep 11 16:16:34.453434 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:34.453379    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t" podStartSLOduration=13.453361658 podStartE2EDuration="13.453361658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:23.447430926 +0000 UTC m=+843.293074481" watchObservedRunningTime="2026-09-11 16:16:34.453361658 +0000 UTC m=+854.299005212"
Sep 11 16:16:57.941662 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:57.941626    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk"]
Sep 11 16:16:57.942123 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:57.941825    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk" podUID="5f67beb5-0a9d-497f-8c64-a9b114757652" containerName="payload-processing" containerID="cri-o://6795952cfc5d61b01a37d7ae9aa17323e8ad3d495addf102f91d7265820696ce" gracePeriod=30
Sep 11 16:16:57.978598 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:57.978564    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t"]
Sep 11 16:16:57.978859 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:57.978833    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t" podUID="eb4f5654-5483-4afb-92b9-38c388417a86" containerName="payload-pre-processing" containerID="cri-o://3235d4a842cddf3cec67ea5407c5059a5ee110e640a2969b4d5f9d531fa14072" gracePeriod=30
Sep 11 16:16:58.273625 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.273602    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk"
Sep 11 16:16:58.276475 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.276456    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t"
Sep 11 16:16:58.315071 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.315023    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f67beb5-0a9d-497f-8c64-a9b114757652-plugins-config-volume\") pod \"5f67beb5-0a9d-497f-8c64-a9b114757652\" (UID: \"5f67beb5-0a9d-497f-8c64-a9b114757652\") "
Sep 11 16:16:58.315243 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.315106    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb4f5654-5483-4afb-92b9-38c388417a86-plugins-config-volume\") pod \"eb4f5654-5483-4afb-92b9-38c388417a86\" (UID: \"eb4f5654-5483-4afb-92b9-38c388417a86\") "
Sep 11 16:16:58.315243 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.315195    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrnc2\" (UniqueName: \"kubernetes.io/projected/eb4f5654-5483-4afb-92b9-38c388417a86-kube-api-access-qrnc2\") pod \"eb4f5654-5483-4afb-92b9-38c388417a86\" (UID: \"eb4f5654-5483-4afb-92b9-38c388417a86\") "
Sep 11 16:16:58.315363 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.315244    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxdjl\" (UniqueName: \"kubernetes.io/projected/5f67beb5-0a9d-497f-8c64-a9b114757652-kube-api-access-hxdjl\") pod \"5f67beb5-0a9d-497f-8c64-a9b114757652\" (UID: \"5f67beb5-0a9d-497f-8c64-a9b114757652\") "
Sep 11 16:16:58.315489 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.315458    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f67beb5-0a9d-497f-8c64-a9b114757652-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5f67beb5-0a9d-497f-8c64-a9b114757652" (UID: "5f67beb5-0a9d-497f-8c64-a9b114757652"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:16:58.315805 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.315486    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb4f5654-5483-4afb-92b9-38c388417a86-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb4f5654-5483-4afb-92b9-38c388417a86" (UID: "eb4f5654-5483-4afb-92b9-38c388417a86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:16:58.326925 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.317880    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4f5654-5483-4afb-92b9-38c388417a86-kube-api-access-qrnc2" (OuterVolumeSpecName: "kube-api-access-qrnc2") pod "eb4f5654-5483-4afb-92b9-38c388417a86" (UID: "eb4f5654-5483-4afb-92b9-38c388417a86"). InnerVolumeSpecName "kube-api-access-qrnc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:16:58.326925 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.325587    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f67beb5-0a9d-497f-8c64-a9b114757652-kube-api-access-hxdjl" (OuterVolumeSpecName: "kube-api-access-hxdjl") pod "5f67beb5-0a9d-497f-8c64-a9b114757652" (UID: "5f67beb5-0a9d-497f-8c64-a9b114757652"). InnerVolumeSpecName "kube-api-access-hxdjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:16:58.416856 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.416823    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb4f5654-5483-4afb-92b9-38c388417a86-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:16:58.416856 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.416852    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrnc2\" (UniqueName: \"kubernetes.io/projected/eb4f5654-5483-4afb-92b9-38c388417a86-kube-api-access-qrnc2\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:16:58.416856 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.416861    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxdjl\" (UniqueName: \"kubernetes.io/projected/5f67beb5-0a9d-497f-8c64-a9b114757652-kube-api-access-hxdjl\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:16:58.417100 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.416870    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f67beb5-0a9d-497f-8c64-a9b114757652-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:16:58.585420 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.585320    2577 generic.go:356] "Generic (PLEG): container finished" podID="5f67beb5-0a9d-497f-8c64-a9b114757652" containerID="6795952cfc5d61b01a37d7ae9aa17323e8ad3d495addf102f91d7265820696ce" exitCode=0
Sep 11 16:16:58.585420 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.585382    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk"
Sep 11 16:16:58.585420 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.585404    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk" event={"ID":"5f67beb5-0a9d-497f-8c64-a9b114757652","Type":"ContainerDied","Data":"6795952cfc5d61b01a37d7ae9aa17323e8ad3d495addf102f91d7265820696ce"}
Sep 11 16:16:58.585690 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.585494    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk" event={"ID":"5f67beb5-0a9d-497f-8c64-a9b114757652","Type":"ContainerDied","Data":"f67ff1d4b82589107a76556fb7494825e0d9f6fd16fa8ad9636a1dbefdca6cd6"}
Sep 11 16:16:58.585690 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.585525    2577 scope.go:117] "RemoveContainer" containerID="6795952cfc5d61b01a37d7ae9aa17323e8ad3d495addf102f91d7265820696ce"
Sep 11 16:16:58.588020 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.587993    2577 generic.go:356] "Generic (PLEG): container finished" podID="eb4f5654-5483-4afb-92b9-38c388417a86" containerID="3235d4a842cddf3cec67ea5407c5059a5ee110e640a2969b4d5f9d531fa14072" exitCode=0
Sep 11 16:16:58.588392 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.588234    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t" event={"ID":"eb4f5654-5483-4afb-92b9-38c388417a86","Type":"ContainerDied","Data":"3235d4a842cddf3cec67ea5407c5059a5ee110e640a2969b4d5f9d531fa14072"}
Sep 11 16:16:58.588568 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.588544    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t" event={"ID":"eb4f5654-5483-4afb-92b9-38c388417a86","Type":"ContainerDied","Data":"54dfedbce43bcfe19005ae76314e4c5d8bb02047c19ed60d46566219dba10e41"}
Sep 11 16:16:58.588692 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.588303    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t"
Sep 11 16:16:58.596219 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.596198    2577 scope.go:117] "RemoveContainer" containerID="6795952cfc5d61b01a37d7ae9aa17323e8ad3d495addf102f91d7265820696ce"
Sep 11 16:16:58.596612 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:16:58.596577    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6795952cfc5d61b01a37d7ae9aa17323e8ad3d495addf102f91d7265820696ce\": container with ID starting with 6795952cfc5d61b01a37d7ae9aa17323e8ad3d495addf102f91d7265820696ce not found: ID does not exist" containerID="6795952cfc5d61b01a37d7ae9aa17323e8ad3d495addf102f91d7265820696ce"
Sep 11 16:16:58.596740 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.596613    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6795952cfc5d61b01a37d7ae9aa17323e8ad3d495addf102f91d7265820696ce"} err="failed to get container status \"6795952cfc5d61b01a37d7ae9aa17323e8ad3d495addf102f91d7265820696ce\": rpc error: code = NotFound desc = could not find container \"6795952cfc5d61b01a37d7ae9aa17323e8ad3d495addf102f91d7265820696ce\": container with ID starting with 6795952cfc5d61b01a37d7ae9aa17323e8ad3d495addf102f91d7265820696ce not found: ID does not exist"
Sep 11 16:16:58.596740 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.596636    2577 scope.go:117] "RemoveContainer" containerID="3235d4a842cddf3cec67ea5407c5059a5ee110e640a2969b4d5f9d531fa14072"
Sep 11 16:16:58.605614 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.605598    2577 scope.go:117] "RemoveContainer" containerID="3235d4a842cddf3cec67ea5407c5059a5ee110e640a2969b4d5f9d531fa14072"
Sep 11 16:16:58.605888 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:16:58.605871    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3235d4a842cddf3cec67ea5407c5059a5ee110e640a2969b4d5f9d531fa14072\": container with ID starting with 3235d4a842cddf3cec67ea5407c5059a5ee110e640a2969b4d5f9d531fa14072 not found: ID does not exist" containerID="3235d4a842cddf3cec67ea5407c5059a5ee110e640a2969b4d5f9d531fa14072"
Sep 11 16:16:58.605962 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.605898    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3235d4a842cddf3cec67ea5407c5059a5ee110e640a2969b4d5f9d531fa14072"} err="failed to get container status \"3235d4a842cddf3cec67ea5407c5059a5ee110e640a2969b4d5f9d531fa14072\": rpc error: code = NotFound desc = could not find container \"3235d4a842cddf3cec67ea5407c5059a5ee110e640a2969b4d5f9d531fa14072\": container with ID starting with 3235d4a842cddf3cec67ea5407c5059a5ee110e640a2969b4d5f9d531fa14072 not found: ID does not exist"
Sep 11 16:16:58.636181 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.636147    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t"]
Sep 11 16:16:58.646541 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.646507    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-02a2f652-554df7f558-xcp7t"]
Sep 11 16:16:58.674252 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.674220    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk"]
Sep 11 16:16:58.691890 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:16:58.691859    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-02a2f652-784dc64dbb-n42pk"]
Sep 11 16:17:00.632940 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:00.632906    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f67beb5-0a9d-497f-8c64-a9b114757652" path="/var/lib/kubelet/pods/5f67beb5-0a9d-497f-8c64-a9b114757652/volumes"
Sep 11 16:17:00.633331 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:00.633274    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb4f5654-5483-4afb-92b9-38c388417a86" path="/var/lib/kubelet/pods/eb4f5654-5483-4afb-92b9-38c388417a86/volumes"
Sep 11 16:17:04.677029 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.676997    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"]
Sep 11 16:17:04.677418 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.677394    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb4f5654-5483-4afb-92b9-38c388417a86" containerName="payload-pre-processing"
Sep 11 16:17:04.677418 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.677408    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb4f5654-5483-4afb-92b9-38c388417a86" containerName="payload-pre-processing"
Sep 11 16:17:04.677418 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.677419    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f67beb5-0a9d-497f-8c64-a9b114757652" containerName="payload-processing"
Sep 11 16:17:04.677520 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.677425    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f67beb5-0a9d-497f-8c64-a9b114757652" containerName="payload-processing"
Sep 11 16:17:04.677520 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.677486    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f67beb5-0a9d-497f-8c64-a9b114757652" containerName="payload-processing"
Sep 11 16:17:04.677520 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.677509    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb4f5654-5483-4afb-92b9-38c388417a86" containerName="payload-pre-processing"
Sep 11 16:17:04.681952 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.681927    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"
Sep 11 16:17:04.685064 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.685046    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-28f629bb-serving-cert\""
Sep 11 16:17:04.702720 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.702690    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"]
Sep 11 16:17:04.775918 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.775882    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/265189f5-bc4b-42ae-805c-29278a77c0f2-metrics-certs\") pod \"maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h\" (UID: \"265189f5-bc4b-42ae-805c-29278a77c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"
Sep 11 16:17:04.776084 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.775956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/265189f5-bc4b-42ae-805c-29278a77c0f2-maas-api-tls\") pod \"maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h\" (UID: \"265189f5-bc4b-42ae-805c-29278a77c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"
Sep 11 16:17:04.776084 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.775990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/265189f5-bc4b-42ae-805c-29278a77c0f2-openshift-service-ca\") pod \"maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h\" (UID: \"265189f5-bc4b-42ae-805c-29278a77c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"
Sep 11 16:17:04.776245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.776142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptkmt\" (UniqueName: \"kubernetes.io/projected/265189f5-bc4b-42ae-805c-29278a77c0f2-kube-api-access-ptkmt\") pod \"maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h\" (UID: \"265189f5-bc4b-42ae-805c-29278a77c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"
Sep 11 16:17:04.877496 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.877452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/265189f5-bc4b-42ae-805c-29278a77c0f2-maas-api-tls\") pod \"maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h\" (UID: \"265189f5-bc4b-42ae-805c-29278a77c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"
Sep 11 16:17:04.877675 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.877566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/265189f5-bc4b-42ae-805c-29278a77c0f2-openshift-service-ca\") pod \"maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h\" (UID: \"265189f5-bc4b-42ae-805c-29278a77c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"
Sep 11 16:17:04.877675 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.877615    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptkmt\" (UniqueName: \"kubernetes.io/projected/265189f5-bc4b-42ae-805c-29278a77c0f2-kube-api-access-ptkmt\") pod \"maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h\" (UID: \"265189f5-bc4b-42ae-805c-29278a77c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"
Sep 11 16:17:04.877760 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.877683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/265189f5-bc4b-42ae-805c-29278a77c0f2-metrics-certs\") pod \"maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h\" (UID: \"265189f5-bc4b-42ae-805c-29278a77c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"
Sep 11 16:17:04.878228 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.878209    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/265189f5-bc4b-42ae-805c-29278a77c0f2-openshift-service-ca\") pod \"maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h\" (UID: \"265189f5-bc4b-42ae-805c-29278a77c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"
Sep 11 16:17:04.880101 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.880079    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/265189f5-bc4b-42ae-805c-29278a77c0f2-metrics-certs\") pod \"maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h\" (UID: \"265189f5-bc4b-42ae-805c-29278a77c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"
Sep 11 16:17:04.880184 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.880101    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/265189f5-bc4b-42ae-805c-29278a77c0f2-maas-api-tls\") pod \"maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h\" (UID: \"265189f5-bc4b-42ae-805c-29278a77c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"
Sep 11 16:17:04.890588 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:04.890557    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptkmt\" (UniqueName: \"kubernetes.io/projected/265189f5-bc4b-42ae-805c-29278a77c0f2-kube-api-access-ptkmt\") pod \"maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h\" (UID: \"265189f5-bc4b-42ae-805c-29278a77c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"
Sep 11 16:17:05.022583 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.022496    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"
Sep 11 16:17:05.309124 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.307730    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"]
Sep 11 16:17:05.573296 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.573211    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf"]
Sep 11 16:17:05.578017 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.577998    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf"
Sep 11 16:17:05.581964 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.581943    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-28f629bb\""
Sep 11 16:17:05.600549 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.600515    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf"]
Sep 11 16:17:05.628192 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.628161    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h" event={"ID":"265189f5-bc4b-42ae-805c-29278a77c0f2","Type":"ContainerStarted","Data":"e235c3be69de590eb2ec58e65e581a7df622efc4fc184b4b951cf28b94fb6f1a"}
Sep 11 16:17:05.685850 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.685810    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztfww\" (UniqueName: \"kubernetes.io/projected/5dd06f1e-9377-4224-be39-17036698db69-kube-api-access-ztfww\") pod \"payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf\" (UID: \"5dd06f1e-9377-4224-be39-17036698db69\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf"
Sep 11 16:17:05.686287 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.685863    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd06f1e-9377-4224-be39-17036698db69-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf\" (UID: \"5dd06f1e-9377-4224-be39-17036698db69\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf"
Sep 11 16:17:05.711873 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.711839    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx"]
Sep 11 16:17:05.715550 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.715522    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx"
Sep 11 16:17:05.728260 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.728235    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-28f629bb-dockercfg-jtsqh\""
Sep 11 16:17:05.758421 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.758386    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx"]
Sep 11 16:17:05.786616 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.786583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztfww\" (UniqueName: \"kubernetes.io/projected/5dd06f1e-9377-4224-be39-17036698db69-kube-api-access-ztfww\") pod \"payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf\" (UID: \"5dd06f1e-9377-4224-be39-17036698db69\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf"
Sep 11 16:17:05.786789 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.786623    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd06f1e-9377-4224-be39-17036698db69-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf\" (UID: \"5dd06f1e-9377-4224-be39-17036698db69\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf"
Sep 11 16:17:05.786789 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.786660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2f93f3-4b3e-45ff-af24-786b95fb0ab2-plugins-config-volume\") pod \"payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx\" (UID: \"9f2f93f3-4b3e-45ff-af24-786b95fb0ab2\") " pod="openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx"
Sep 11 16:17:05.786789 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.786679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxhx6\" (UniqueName: \"kubernetes.io/projected/9f2f93f3-4b3e-45ff-af24-786b95fb0ab2-kube-api-access-fxhx6\") pod \"payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx\" (UID: \"9f2f93f3-4b3e-45ff-af24-786b95fb0ab2\") " pod="openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx"
Sep 11 16:17:05.787365 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.787343    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd06f1e-9377-4224-be39-17036698db69-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf\" (UID: \"5dd06f1e-9377-4224-be39-17036698db69\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf"
Sep 11 16:17:05.798472 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.798446    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztfww\" (UniqueName: \"kubernetes.io/projected/5dd06f1e-9377-4224-be39-17036698db69-kube-api-access-ztfww\") pod \"payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf\" (UID: \"5dd06f1e-9377-4224-be39-17036698db69\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf"
Sep 11 16:17:05.887166 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.887124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2f93f3-4b3e-45ff-af24-786b95fb0ab2-plugins-config-volume\") pod \"payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx\" (UID: \"9f2f93f3-4b3e-45ff-af24-786b95fb0ab2\") " pod="openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx"
Sep 11 16:17:05.887310 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.887168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxhx6\" (UniqueName: \"kubernetes.io/projected/9f2f93f3-4b3e-45ff-af24-786b95fb0ab2-kube-api-access-fxhx6\") pod \"payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx\" (UID: \"9f2f93f3-4b3e-45ff-af24-786b95fb0ab2\") " pod="openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx"
Sep 11 16:17:05.887780 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.887759    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2f93f3-4b3e-45ff-af24-786b95fb0ab2-plugins-config-volume\") pod \"payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx\" (UID: \"9f2f93f3-4b3e-45ff-af24-786b95fb0ab2\") " pod="openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx"
Sep 11 16:17:05.889836 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.889817    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf"
Sep 11 16:17:05.905163 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:05.905127    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxhx6\" (UniqueName: \"kubernetes.io/projected/9f2f93f3-4b3e-45ff-af24-786b95fb0ab2-kube-api-access-fxhx6\") pod \"payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx\" (UID: \"9f2f93f3-4b3e-45ff-af24-786b95fb0ab2\") " pod="openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx"
Sep 11 16:17:06.030259 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:06.030215    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx"
Sep 11 16:17:06.192102 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:06.192069    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf"]
Sep 11 16:17:06.194536 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:17:06.194503    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd06f1e_9377_4224_be39_17036698db69.slice/crio-f5b084bbff02d056ee8633e0939e1533215828484a2d0e88714dda7b046ba25b WatchSource:0}: Error finding container f5b084bbff02d056ee8633e0939e1533215828484a2d0e88714dda7b046ba25b: Status 404 returned error can't find the container with id f5b084bbff02d056ee8633e0939e1533215828484a2d0e88714dda7b046ba25b
Sep 11 16:17:06.332750 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:17:06.332714    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2f93f3_4b3e_45ff_af24_786b95fb0ab2.slice/crio-6ef0828daa17f9042b6ec7869ae04703f05577009c4ed786efc71fff3a4ce9cf WatchSource:0}: Error finding container 6ef0828daa17f9042b6ec7869ae04703f05577009c4ed786efc71fff3a4ce9cf: Status 404 returned error can't find the container with id 6ef0828daa17f9042b6ec7869ae04703f05577009c4ed786efc71fff3a4ce9cf
Sep 11 16:17:06.335016 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:06.334994    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx"]
Sep 11 16:17:06.636528 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:06.636487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx" event={"ID":"9f2f93f3-4b3e-45ff-af24-786b95fb0ab2","Type":"ContainerStarted","Data":"d5b642a513dad92171d512097b930659095deeb4d047e281c7e21a2af5074cea"}
Sep 11 16:17:06.636724 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:06.636547    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx"
Sep 11 16:17:06.636724 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:06.636574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx" event={"ID":"9f2f93f3-4b3e-45ff-af24-786b95fb0ab2","Type":"ContainerStarted","Data":"6ef0828daa17f9042b6ec7869ae04703f05577009c4ed786efc71fff3a4ce9cf"}
Sep 11 16:17:06.638332 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:06.638309    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf" event={"ID":"5dd06f1e-9377-4224-be39-17036698db69","Type":"ContainerStarted","Data":"1ec7320ef93f056e8a707969efa969e8045fe7a191e46440cf369ccccd747774"}
Sep 11 16:17:06.638469 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:06.638350    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf" event={"ID":"5dd06f1e-9377-4224-be39-17036698db69","Type":"ContainerStarted","Data":"f5b084bbff02d056ee8633e0939e1533215828484a2d0e88714dda7b046ba25b"}
Sep 11 16:17:06.638469 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:06.638372    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf"
Sep 11 16:17:06.640337 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:06.640313    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h" event={"ID":"265189f5-bc4b-42ae-805c-29278a77c0f2","Type":"ContainerStarted","Data":"c18acbf1d96e1318418a8db1716b5283ef43072bded0d0a29c3f27a047612aef"}
Sep 11 16:17:06.640482 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:06.640467    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"
Sep 11 16:17:06.674519 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:06.674470    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx" podStartSLOduration=1.674452492 podStartE2EDuration="1.674452492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:06.673429457 +0000 UTC m=+886.519073012" watchObservedRunningTime="2026-09-11 16:17:06.674452492 +0000 UTC m=+886.520096044"
Sep 11 16:17:06.727537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:06.727476    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h" podStartSLOduration=2.196925162 podStartE2EDuration="2.727458924s" totalImagesPullingTime="530.533762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:06.726536706 +0000 UTC m=+886.572180262" watchObservedRunningTime="2026-09-11 16:17:06.727458924 +0000 UTC m=+886.573102479"
Sep 11 16:17:06.757355 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:06.757290    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf" podStartSLOduration=1.757268667 podStartE2EDuration="1.757268667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:06.755361568 +0000 UTC m=+886.601005122" watchObservedRunningTime="2026-09-11 16:17:06.757268667 +0000 UTC m=+886.602912223"
Sep 11 16:17:12.652153 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:12.652077    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"
Sep 11 16:17:17.649536 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:17.649506    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx"
Sep 11 16:17:17.649966 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:17.649588    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf"
Sep 11 16:17:20.566782 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:20.566753    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/1.log"
Sep 11 16:17:20.567222 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:20.567010    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/1.log"
Sep 11 16:17:20.573687 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:20.573654    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ng2ww_07d57c04-0697-419b-9d3e-ca11761da85c/ovn-acl-logging/0.log"
Sep 11 16:17:20.574097 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:20.574080    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ng2ww_07d57c04-0697-419b-9d3e-ca11761da85c/ovn-acl-logging/0.log"
Sep 11 16:17:35.089450 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.089409    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"]
Sep 11 16:17:35.089873 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.089609    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h" podUID="265189f5-bc4b-42ae-805c-29278a77c0f2" containerName="maas-api" containerID="cri-o://c18acbf1d96e1318418a8db1716b5283ef43072bded0d0a29c3f27a047612aef" gracePeriod=30
Sep 11 16:17:35.183066 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.183017    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx"]
Sep 11 16:17:35.183338 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.183306    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx" podUID="9f2f93f3-4b3e-45ff-af24-786b95fb0ab2" containerName="payload-processing" containerID="cri-o://d5b642a513dad92171d512097b930659095deeb4d047e281c7e21a2af5074cea" gracePeriod=30
Sep 11 16:17:35.195554 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.195523    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf"]
Sep 11 16:17:35.196023 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.195966    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf" podUID="5dd06f1e-9377-4224-be39-17036698db69" containerName="payload-pre-processing" containerID="cri-o://1ec7320ef93f056e8a707969efa969e8045fe7a191e46440cf369ccccd747774" gracePeriod=30
Sep 11 16:17:35.373178 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.372903    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"
Sep 11 16:17:35.444704 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.444676    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx"
Sep 11 16:17:35.468855 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.468820    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/265189f5-bc4b-42ae-805c-29278a77c0f2-maas-api-tls\") pod \"265189f5-bc4b-42ae-805c-29278a77c0f2\" (UID: \"265189f5-bc4b-42ae-805c-29278a77c0f2\") "
Sep 11 16:17:35.469672 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.468889    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptkmt\" (UniqueName: \"kubernetes.io/projected/265189f5-bc4b-42ae-805c-29278a77c0f2-kube-api-access-ptkmt\") pod \"265189f5-bc4b-42ae-805c-29278a77c0f2\" (UID: \"265189f5-bc4b-42ae-805c-29278a77c0f2\") "
Sep 11 16:17:35.469672 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.468924    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/265189f5-bc4b-42ae-805c-29278a77c0f2-openshift-service-ca\") pod \"265189f5-bc4b-42ae-805c-29278a77c0f2\" (UID: \"265189f5-bc4b-42ae-805c-29278a77c0f2\") "
Sep 11 16:17:35.469672 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.469093    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/265189f5-bc4b-42ae-805c-29278a77c0f2-metrics-certs\") pod \"265189f5-bc4b-42ae-805c-29278a77c0f2\" (UID: \"265189f5-bc4b-42ae-805c-29278a77c0f2\") "
Sep 11 16:17:35.469672 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.469504    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/265189f5-bc4b-42ae-805c-29278a77c0f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "265189f5-bc4b-42ae-805c-29278a77c0f2" (UID: "265189f5-bc4b-42ae-805c-29278a77c0f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:17:35.470987 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.470818    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/265189f5-bc4b-42ae-805c-29278a77c0f2-openshift-service-ca\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:17:35.471522 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.471500    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/265189f5-bc4b-42ae-805c-29278a77c0f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "265189f5-bc4b-42ae-805c-29278a77c0f2" (UID: "265189f5-bc4b-42ae-805c-29278a77c0f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:17:35.471890 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.471867    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/265189f5-bc4b-42ae-805c-29278a77c0f2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "265189f5-bc4b-42ae-805c-29278a77c0f2" (UID: "265189f5-bc4b-42ae-805c-29278a77c0f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:17:35.473571 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.473541    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265189f5-bc4b-42ae-805c-29278a77c0f2-kube-api-access-ptkmt" (OuterVolumeSpecName: "kube-api-access-ptkmt") pod "265189f5-bc4b-42ae-805c-29278a77c0f2" (UID: "265189f5-bc4b-42ae-805c-29278a77c0f2"). InnerVolumeSpecName "kube-api-access-ptkmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:17:35.487102 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.487070    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf"
Sep 11 16:17:35.571659 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.571615    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2f93f3-4b3e-45ff-af24-786b95fb0ab2-plugins-config-volume\") pod \"9f2f93f3-4b3e-45ff-af24-786b95fb0ab2\" (UID: \"9f2f93f3-4b3e-45ff-af24-786b95fb0ab2\") "
Sep 11 16:17:35.571830 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.571684    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxhx6\" (UniqueName: \"kubernetes.io/projected/9f2f93f3-4b3e-45ff-af24-786b95fb0ab2-kube-api-access-fxhx6\") pod \"9f2f93f3-4b3e-45ff-af24-786b95fb0ab2\" (UID: \"9f2f93f3-4b3e-45ff-af24-786b95fb0ab2\") "
Sep 11 16:17:35.571925 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.571906    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/265189f5-bc4b-42ae-805c-29278a77c0f2-metrics-certs\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:17:35.571963 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.571931    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/265189f5-bc4b-42ae-805c-29278a77c0f2-maas-api-tls\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:17:35.571963 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.571946    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptkmt\" (UniqueName: \"kubernetes.io/projected/265189f5-bc4b-42ae-805c-29278a77c0f2-kube-api-access-ptkmt\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:17:35.572080 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.572021    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2f93f3-4b3e-45ff-af24-786b95fb0ab2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f2f93f3-4b3e-45ff-af24-786b95fb0ab2" (UID: "9f2f93f3-4b3e-45ff-af24-786b95fb0ab2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:17:35.574075 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.574052    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2f93f3-4b3e-45ff-af24-786b95fb0ab2-kube-api-access-fxhx6" (OuterVolumeSpecName: "kube-api-access-fxhx6") pod "9f2f93f3-4b3e-45ff-af24-786b95fb0ab2" (UID: "9f2f93f3-4b3e-45ff-af24-786b95fb0ab2"). InnerVolumeSpecName "kube-api-access-fxhx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:17:35.672893 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.672855    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztfww\" (UniqueName: \"kubernetes.io/projected/5dd06f1e-9377-4224-be39-17036698db69-kube-api-access-ztfww\") pod \"5dd06f1e-9377-4224-be39-17036698db69\" (UID: \"5dd06f1e-9377-4224-be39-17036698db69\") "
Sep 11 16:17:35.673134 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.672932    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd06f1e-9377-4224-be39-17036698db69-plugins-config-volume\") pod \"5dd06f1e-9377-4224-be39-17036698db69\" (UID: \"5dd06f1e-9377-4224-be39-17036698db69\") "
Sep 11 16:17:35.673134 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.673113    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxhx6\" (UniqueName: \"kubernetes.io/projected/9f2f93f3-4b3e-45ff-af24-786b95fb0ab2-kube-api-access-fxhx6\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:17:35.673134 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.673126    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2f93f3-4b3e-45ff-af24-786b95fb0ab2-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:17:35.673426 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.673399    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd06f1e-9377-4224-be39-17036698db69-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5dd06f1e-9377-4224-be39-17036698db69" (UID: "5dd06f1e-9377-4224-be39-17036698db69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:17:35.675183 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.675160    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd06f1e-9377-4224-be39-17036698db69-kube-api-access-ztfww" (OuterVolumeSpecName: "kube-api-access-ztfww") pod "5dd06f1e-9377-4224-be39-17036698db69" (UID: "5dd06f1e-9377-4224-be39-17036698db69"). InnerVolumeSpecName "kube-api-access-ztfww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:17:35.774520 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.774476    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd06f1e-9377-4224-be39-17036698db69-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:17:35.774520 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.774508    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztfww\" (UniqueName: \"kubernetes.io/projected/5dd06f1e-9377-4224-be39-17036698db69-kube-api-access-ztfww\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:17:35.794879 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.794841    2577 generic.go:356] "Generic (PLEG): container finished" podID="265189f5-bc4b-42ae-805c-29278a77c0f2" containerID="c18acbf1d96e1318418a8db1716b5283ef43072bded0d0a29c3f27a047612aef" exitCode=0
Sep 11 16:17:35.795015 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.794890    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"
Sep 11 16:17:35.795015 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.794938    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h" event={"ID":"265189f5-bc4b-42ae-805c-29278a77c0f2","Type":"ContainerDied","Data":"c18acbf1d96e1318418a8db1716b5283ef43072bded0d0a29c3f27a047612aef"}
Sep 11 16:17:35.795015 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.794986    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h" event={"ID":"265189f5-bc4b-42ae-805c-29278a77c0f2","Type":"ContainerDied","Data":"e235c3be69de590eb2ec58e65e581a7df622efc4fc184b4b951cf28b94fb6f1a"}
Sep 11 16:17:35.795188 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.795017    2577 scope.go:117] "RemoveContainer" containerID="c18acbf1d96e1318418a8db1716b5283ef43072bded0d0a29c3f27a047612aef"
Sep 11 16:17:35.796313 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.796293    2577 generic.go:356] "Generic (PLEG): container finished" podID="9f2f93f3-4b3e-45ff-af24-786b95fb0ab2" containerID="d5b642a513dad92171d512097b930659095deeb4d047e281c7e21a2af5074cea" exitCode=0
Sep 11 16:17:35.796404 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.796341    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx"
Sep 11 16:17:35.796404 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.796373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx" event={"ID":"9f2f93f3-4b3e-45ff-af24-786b95fb0ab2","Type":"ContainerDied","Data":"d5b642a513dad92171d512097b930659095deeb4d047e281c7e21a2af5074cea"}
Sep 11 16:17:35.796570 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.796419    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx" event={"ID":"9f2f93f3-4b3e-45ff-af24-786b95fb0ab2","Type":"ContainerDied","Data":"6ef0828daa17f9042b6ec7869ae04703f05577009c4ed786efc71fff3a4ce9cf"}
Sep 11 16:17:35.805207 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.805161    2577 generic.go:356] "Generic (PLEG): container finished" podID="5dd06f1e-9377-4224-be39-17036698db69" containerID="1ec7320ef93f056e8a707969efa969e8045fe7a191e46440cf369ccccd747774" exitCode=0
Sep 11 16:17:35.805372 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.805237    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf" event={"ID":"5dd06f1e-9377-4224-be39-17036698db69","Type":"ContainerDied","Data":"1ec7320ef93f056e8a707969efa969e8045fe7a191e46440cf369ccccd747774"}
Sep 11 16:17:35.805372 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.805340    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf" event={"ID":"5dd06f1e-9377-4224-be39-17036698db69","Type":"ContainerDied","Data":"f5b084bbff02d056ee8633e0939e1533215828484a2d0e88714dda7b046ba25b"}
Sep 11 16:17:35.805563 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.805539    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf"
Sep 11 16:17:35.811296 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.811268    2577 scope.go:117] "RemoveContainer" containerID="c18acbf1d96e1318418a8db1716b5283ef43072bded0d0a29c3f27a047612aef"
Sep 11 16:17:35.811686 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:17:35.811659    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c18acbf1d96e1318418a8db1716b5283ef43072bded0d0a29c3f27a047612aef\": container with ID starting with c18acbf1d96e1318418a8db1716b5283ef43072bded0d0a29c3f27a047612aef not found: ID does not exist" containerID="c18acbf1d96e1318418a8db1716b5283ef43072bded0d0a29c3f27a047612aef"
Sep 11 16:17:35.811746 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.811697    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c18acbf1d96e1318418a8db1716b5283ef43072bded0d0a29c3f27a047612aef"} err="failed to get container status \"c18acbf1d96e1318418a8db1716b5283ef43072bded0d0a29c3f27a047612aef\": rpc error: code = NotFound desc = could not find container \"c18acbf1d96e1318418a8db1716b5283ef43072bded0d0a29c3f27a047612aef\": container with ID starting with c18acbf1d96e1318418a8db1716b5283ef43072bded0d0a29c3f27a047612aef not found: ID does not exist"
Sep 11 16:17:35.811746 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.811720    2577 scope.go:117] "RemoveContainer" containerID="d5b642a513dad92171d512097b930659095deeb4d047e281c7e21a2af5074cea"
Sep 11 16:17:35.821192 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.821161    2577 scope.go:117] "RemoveContainer" containerID="d5b642a513dad92171d512097b930659095deeb4d047e281c7e21a2af5074cea"
Sep 11 16:17:35.821601 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:17:35.821581    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b642a513dad92171d512097b930659095deeb4d047e281c7e21a2af5074cea\": container with ID starting with d5b642a513dad92171d512097b930659095deeb4d047e281c7e21a2af5074cea not found: ID does not exist" containerID="d5b642a513dad92171d512097b930659095deeb4d047e281c7e21a2af5074cea"
Sep 11 16:17:35.821654 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.821612    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b642a513dad92171d512097b930659095deeb4d047e281c7e21a2af5074cea"} err="failed to get container status \"d5b642a513dad92171d512097b930659095deeb4d047e281c7e21a2af5074cea\": rpc error: code = NotFound desc = could not find container \"d5b642a513dad92171d512097b930659095deeb4d047e281c7e21a2af5074cea\": container with ID starting with d5b642a513dad92171d512097b930659095deeb4d047e281c7e21a2af5074cea not found: ID does not exist"
Sep 11 16:17:35.821654 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.821632    2577 scope.go:117] "RemoveContainer" containerID="1ec7320ef93f056e8a707969efa969e8045fe7a191e46440cf369ccccd747774"
Sep 11 16:17:35.830509 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.830487    2577 scope.go:117] "RemoveContainer" containerID="1ec7320ef93f056e8a707969efa969e8045fe7a191e46440cf369ccccd747774"
Sep 11 16:17:35.830838 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:17:35.830818    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ec7320ef93f056e8a707969efa969e8045fe7a191e46440cf369ccccd747774\": container with ID starting with 1ec7320ef93f056e8a707969efa969e8045fe7a191e46440cf369ccccd747774 not found: ID does not exist" containerID="1ec7320ef93f056e8a707969efa969e8045fe7a191e46440cf369ccccd747774"
Sep 11 16:17:35.830900 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.830847    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ec7320ef93f056e8a707969efa969e8045fe7a191e46440cf369ccccd747774"} err="failed to get container status \"1ec7320ef93f056e8a707969efa969e8045fe7a191e46440cf369ccccd747774\": rpc error: code = NotFound desc = could not find container \"1ec7320ef93f056e8a707969efa969e8045fe7a191e46440cf369ccccd747774\": container with ID starting with 1ec7320ef93f056e8a707969efa969e8045fe7a191e46440cf369ccccd747774 not found: ID does not exist"
Sep 11 16:17:35.860476 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.860441    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"]
Sep 11 16:17:35.879410 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.879380    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-28f629bb-5694fd5fdc-8mr8h"]
Sep 11 16:17:35.913292 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.913254    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx"]
Sep 11 16:17:35.921110 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.921075    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-28f629bb-79d5565ffc-gdhxx"]
Sep 11 16:17:35.935367 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.935280    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf"]
Sep 11 16:17:35.938967 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:35.938939    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-28f629bb-7b865fbd46-2s7gf"]
Sep 11 16:17:36.632943 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:36.632903    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="265189f5-bc4b-42ae-805c-29278a77c0f2" path="/var/lib/kubelet/pods/265189f5-bc4b-42ae-805c-29278a77c0f2/volumes"
Sep 11 16:17:36.633407 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:36.633286    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dd06f1e-9377-4224-be39-17036698db69" path="/var/lib/kubelet/pods/5dd06f1e-9377-4224-be39-17036698db69/volumes"
Sep 11 16:17:36.633593 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:36.633580    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f2f93f3-4b3e-45ff-af24-786b95fb0ab2" path="/var/lib/kubelet/pods/9f2f93f3-4b3e-45ff-af24-786b95fb0ab2/volumes"
Sep 11 16:17:45.258531 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.258496    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"]
Sep 11 16:17:45.258904 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.258887    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f2f93f3-4b3e-45ff-af24-786b95fb0ab2" containerName="payload-processing"
Sep 11 16:17:45.258904 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.258898    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f2f93f3-4b3e-45ff-af24-786b95fb0ab2" containerName="payload-processing"
Sep 11 16:17:45.258982 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.258909    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="265189f5-bc4b-42ae-805c-29278a77c0f2" containerName="maas-api"
Sep 11 16:17:45.258982 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.258915    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="265189f5-bc4b-42ae-805c-29278a77c0f2" containerName="maas-api"
Sep 11 16:17:45.258982 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.258923    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd06f1e-9377-4224-be39-17036698db69" containerName="payload-pre-processing"
Sep 11 16:17:45.258982 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.258928    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd06f1e-9377-4224-be39-17036698db69" containerName="payload-pre-processing"
Sep 11 16:17:45.259234 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.258987    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="265189f5-bc4b-42ae-805c-29278a77c0f2" containerName="maas-api"
Sep 11 16:17:45.259234 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.258996    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dd06f1e-9377-4224-be39-17036698db69" containerName="payload-pre-processing"
Sep 11 16:17:45.259234 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.259002    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f2f93f3-4b3e-45ff-af24-786b95fb0ab2" containerName="payload-processing"
Sep 11 16:17:45.263757 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.263735    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"
Sep 11 16:17:45.266825 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.266802    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a9b1abbc-serving-cert\""
Sep 11 16:17:45.295625 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.295589    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"]
Sep 11 16:17:45.360739 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.360700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecbaebf9-cb27-42d2-8048-12f457a7a449-maas-api-tls\") pod \"maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr\" (UID: \"ecbaebf9-cb27-42d2-8048-12f457a7a449\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"
Sep 11 16:17:45.360739 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.360741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecbaebf9-cb27-42d2-8048-12f457a7a449-openshift-service-ca\") pod \"maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr\" (UID: \"ecbaebf9-cb27-42d2-8048-12f457a7a449\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"
Sep 11 16:17:45.360970 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.360816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecbaebf9-cb27-42d2-8048-12f457a7a449-metrics-certs\") pod \"maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr\" (UID: \"ecbaebf9-cb27-42d2-8048-12f457a7a449\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"
Sep 11 16:17:45.360970 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.360869    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nxkf\" (UniqueName: \"kubernetes.io/projected/ecbaebf9-cb27-42d2-8048-12f457a7a449-kube-api-access-2nxkf\") pod \"maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr\" (UID: \"ecbaebf9-cb27-42d2-8048-12f457a7a449\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"
Sep 11 16:17:45.462306 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.462270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nxkf\" (UniqueName: \"kubernetes.io/projected/ecbaebf9-cb27-42d2-8048-12f457a7a449-kube-api-access-2nxkf\") pod \"maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr\" (UID: \"ecbaebf9-cb27-42d2-8048-12f457a7a449\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"
Sep 11 16:17:45.462525 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.462316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecbaebf9-cb27-42d2-8048-12f457a7a449-maas-api-tls\") pod \"maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr\" (UID: \"ecbaebf9-cb27-42d2-8048-12f457a7a449\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"
Sep 11 16:17:45.462525 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.462343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecbaebf9-cb27-42d2-8048-12f457a7a449-openshift-service-ca\") pod \"maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr\" (UID: \"ecbaebf9-cb27-42d2-8048-12f457a7a449\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"
Sep 11 16:17:45.462525 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.462372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecbaebf9-cb27-42d2-8048-12f457a7a449-metrics-certs\") pod \"maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr\" (UID: \"ecbaebf9-cb27-42d2-8048-12f457a7a449\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"
Sep 11 16:17:45.463011 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.462987    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecbaebf9-cb27-42d2-8048-12f457a7a449-openshift-service-ca\") pod \"maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr\" (UID: \"ecbaebf9-cb27-42d2-8048-12f457a7a449\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"
Sep 11 16:17:45.464954 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.464935    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecbaebf9-cb27-42d2-8048-12f457a7a449-maas-api-tls\") pod \"maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr\" (UID: \"ecbaebf9-cb27-42d2-8048-12f457a7a449\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"
Sep 11 16:17:45.465068 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.464966    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecbaebf9-cb27-42d2-8048-12f457a7a449-metrics-certs\") pod \"maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr\" (UID: \"ecbaebf9-cb27-42d2-8048-12f457a7a449\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"
Sep 11 16:17:45.474397 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.474368    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nxkf\" (UniqueName: \"kubernetes.io/projected/ecbaebf9-cb27-42d2-8048-12f457a7a449-kube-api-access-2nxkf\") pod \"maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr\" (UID: \"ecbaebf9-cb27-42d2-8048-12f457a7a449\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"
Sep 11 16:17:45.573865 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.573773    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"
Sep 11 16:17:45.861051 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:45.861012    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"]
Sep 11 16:17:45.863502 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:17:45.863474    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbaebf9_cb27_42d2_8048_12f457a7a449.slice/crio-675ace26d30c91a395ac2edc5f0f98e318664e96d626d91b5492008334be13cf WatchSource:0}: Error finding container 675ace26d30c91a395ac2edc5f0f98e318664e96d626d91b5492008334be13cf: Status 404 returned error can't find the container with id 675ace26d30c91a395ac2edc5f0f98e318664e96d626d91b5492008334be13cf
Sep 11 16:17:46.218681 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:46.218604    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm"]
Sep 11 16:17:46.222482 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:46.222458    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm"
Sep 11 16:17:46.233357 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:46.233318    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a9b1abbc\""
Sep 11 16:17:46.233504 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:46.233476    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a9b1abbc-dockercfg-j7r2m\""
Sep 11 16:17:46.241934 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:46.241906    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm"]
Sep 11 16:17:46.267351 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:46.267314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04dec036-ded6-482a-9193-1d3b8b60e9be-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm\" (UID: \"04dec036-ded6-482a-9193-1d3b8b60e9be\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm"
Sep 11 16:17:46.267763 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:46.267408    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2hxp\" (UniqueName: \"kubernetes.io/projected/04dec036-ded6-482a-9193-1d3b8b60e9be-kube-api-access-t2hxp\") pod \"payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm\" (UID: \"04dec036-ded6-482a-9193-1d3b8b60e9be\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm"
Sep 11 16:17:46.368722 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:46.368680    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04dec036-ded6-482a-9193-1d3b8b60e9be-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm\" (UID: \"04dec036-ded6-482a-9193-1d3b8b60e9be\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm"
Sep 11 16:17:46.368886 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:46.368828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2hxp\" (UniqueName: \"kubernetes.io/projected/04dec036-ded6-482a-9193-1d3b8b60e9be-kube-api-access-t2hxp\") pod \"payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm\" (UID: \"04dec036-ded6-482a-9193-1d3b8b60e9be\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm"
Sep 11 16:17:46.369390 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:46.369371    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04dec036-ded6-482a-9193-1d3b8b60e9be-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm\" (UID: \"04dec036-ded6-482a-9193-1d3b8b60e9be\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm"
Sep 11 16:17:46.388755 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:46.388720    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2hxp\" (UniqueName: \"kubernetes.io/projected/04dec036-ded6-482a-9193-1d3b8b60e9be-kube-api-access-t2hxp\") pod \"payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm\" (UID: \"04dec036-ded6-482a-9193-1d3b8b60e9be\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm"
Sep 11 16:17:46.533962 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:46.533865    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm"
Sep 11 16:17:46.820279 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:46.820249    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm"]
Sep 11 16:17:46.821846 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:17:46.821816    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04dec036_ded6_482a_9193_1d3b8b60e9be.slice/crio-5402c44f59280216d1f4aa1d659f4f42c66208b5146e8cf3805d8da61f700fbc WatchSource:0}: Error finding container 5402c44f59280216d1f4aa1d659f4f42c66208b5146e8cf3805d8da61f700fbc: Status 404 returned error can't find the container with id 5402c44f59280216d1f4aa1d659f4f42c66208b5146e8cf3805d8da61f700fbc
Sep 11 16:17:46.869195 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:46.869157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm" event={"ID":"04dec036-ded6-482a-9193-1d3b8b60e9be","Type":"ContainerStarted","Data":"5402c44f59280216d1f4aa1d659f4f42c66208b5146e8cf3805d8da61f700fbc"}
Sep 11 16:17:46.871207 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:46.871171    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr" event={"ID":"ecbaebf9-cb27-42d2-8048-12f457a7a449","Type":"ContainerStarted","Data":"f2e9f7aa12750d5d2c08b26d17dc121fe67c801286105c12b447103f778c3d0d"}
Sep 11 16:17:46.871341 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:46.871214    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr" event={"ID":"ecbaebf9-cb27-42d2-8048-12f457a7a449","Type":"ContainerStarted","Data":"675ace26d30c91a395ac2edc5f0f98e318664e96d626d91b5492008334be13cf"}
Sep 11 16:17:46.871341 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:46.871326    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"
Sep 11 16:17:46.906582 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:46.906516    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr" podStartSLOduration=1.478013719 podStartE2EDuration="1.90649014s" totalImagesPullingTime="428.476421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:46.90531852 +0000 UTC m=+926.750962075" watchObservedRunningTime="2026-09-11 16:17:46.90649014 +0000 UTC m=+926.752133695"
Sep 11 16:17:47.879206 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:47.879169    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm" event={"ID":"04dec036-ded6-482a-9193-1d3b8b60e9be","Type":"ContainerStarted","Data":"3a99c57ba018f79386849bdfbfef852d06f0af3b9ce7956224e0278594a6b71a"}
Sep 11 16:17:47.879687 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:47.879326    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm"
Sep 11 16:17:52.884484 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:52.884458    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"
Sep 11 16:17:52.907975 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:52.907914    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm" podStartSLOduration=6.907898027 podStartE2EDuration="6.907898027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:47.913288508 +0000 UTC m=+927.758932066" watchObservedRunningTime="2026-09-11 16:17:52.907898027 +0000 UTC m=+932.753541584"
Sep 11 16:17:58.886464 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:17:58.886433    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm"
Sep 11 16:18:21.936395 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:21.936348    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"]
Sep 11 16:18:21.936845 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:21.936604    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr" podUID="ecbaebf9-cb27-42d2-8048-12f457a7a449" containerName="maas-api" containerID="cri-o://f2e9f7aa12750d5d2c08b26d17dc121fe67c801286105c12b447103f778c3d0d" gracePeriod=30
Sep 11 16:18:22.026899 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.026853    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm"]
Sep 11 16:18:22.027119 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.027089    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm" podUID="04dec036-ded6-482a-9193-1d3b8b60e9be" containerName="payload-pre-processing" containerID="cri-o://3a99c57ba018f79386849bdfbfef852d06f0af3b9ce7956224e0278594a6b71a" gracePeriod=30
Sep 11 16:18:22.065528 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.065492    2577 generic.go:356] "Generic (PLEG): container finished" podID="ecbaebf9-cb27-42d2-8048-12f457a7a449" containerID="f2e9f7aa12750d5d2c08b26d17dc121fe67c801286105c12b447103f778c3d0d" exitCode=0
Sep 11 16:18:22.065680 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.065548    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr" event={"ID":"ecbaebf9-cb27-42d2-8048-12f457a7a449","Type":"ContainerDied","Data":"f2e9f7aa12750d5d2c08b26d17dc121fe67c801286105c12b447103f778c3d0d"}
Sep 11 16:18:22.274770 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.274744    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"
Sep 11 16:18:22.304788 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.304764    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm"
Sep 11 16:18:22.384900 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.384861    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecbaebf9-cb27-42d2-8048-12f457a7a449-maas-api-tls\") pod \"ecbaebf9-cb27-42d2-8048-12f457a7a449\" (UID: \"ecbaebf9-cb27-42d2-8048-12f457a7a449\") "
Sep 11 16:18:22.385118 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.384929    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nxkf\" (UniqueName: \"kubernetes.io/projected/ecbaebf9-cb27-42d2-8048-12f457a7a449-kube-api-access-2nxkf\") pod \"ecbaebf9-cb27-42d2-8048-12f457a7a449\" (UID: \"ecbaebf9-cb27-42d2-8048-12f457a7a449\") "
Sep 11 16:18:22.385118 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.384962    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04dec036-ded6-482a-9193-1d3b8b60e9be-plugins-config-volume\") pod \"04dec036-ded6-482a-9193-1d3b8b60e9be\" (UID: \"04dec036-ded6-482a-9193-1d3b8b60e9be\") "
Sep 11 16:18:22.385118 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.385006    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecbaebf9-cb27-42d2-8048-12f457a7a449-metrics-certs\") pod \"ecbaebf9-cb27-42d2-8048-12f457a7a449\" (UID: \"ecbaebf9-cb27-42d2-8048-12f457a7a449\") "
Sep 11 16:18:22.385118 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.385082    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecbaebf9-cb27-42d2-8048-12f457a7a449-openshift-service-ca\") pod \"ecbaebf9-cb27-42d2-8048-12f457a7a449\" (UID: \"ecbaebf9-cb27-42d2-8048-12f457a7a449\") "
Sep 11 16:18:22.385353 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.385161    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2hxp\" (UniqueName: \"kubernetes.io/projected/04dec036-ded6-482a-9193-1d3b8b60e9be-kube-api-access-t2hxp\") pod \"04dec036-ded6-482a-9193-1d3b8b60e9be\" (UID: \"04dec036-ded6-482a-9193-1d3b8b60e9be\") "
Sep 11 16:18:22.386163 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.386100    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecbaebf9-cb27-42d2-8048-12f457a7a449-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ecbaebf9-cb27-42d2-8048-12f457a7a449" (UID: "ecbaebf9-cb27-42d2-8048-12f457a7a449"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:18:22.388462 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.388437    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04dec036-ded6-482a-9193-1d3b8b60e9be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "04dec036-ded6-482a-9193-1d3b8b60e9be" (UID: "04dec036-ded6-482a-9193-1d3b8b60e9be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:18:22.389885 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.389858    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04dec036-ded6-482a-9193-1d3b8b60e9be-kube-api-access-t2hxp" (OuterVolumeSpecName: "kube-api-access-t2hxp") pod "04dec036-ded6-482a-9193-1d3b8b60e9be" (UID: "04dec036-ded6-482a-9193-1d3b8b60e9be"). InnerVolumeSpecName "kube-api-access-t2hxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:18:22.391096 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.391068    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecbaebf9-cb27-42d2-8048-12f457a7a449-kube-api-access-2nxkf" (OuterVolumeSpecName: "kube-api-access-2nxkf") pod "ecbaebf9-cb27-42d2-8048-12f457a7a449" (UID: "ecbaebf9-cb27-42d2-8048-12f457a7a449"). InnerVolumeSpecName "kube-api-access-2nxkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:18:22.392498 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.392463    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecbaebf9-cb27-42d2-8048-12f457a7a449-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ecbaebf9-cb27-42d2-8048-12f457a7a449" (UID: "ecbaebf9-cb27-42d2-8048-12f457a7a449"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:18:22.392581 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.392492    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecbaebf9-cb27-42d2-8048-12f457a7a449-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ecbaebf9-cb27-42d2-8048-12f457a7a449" (UID: "ecbaebf9-cb27-42d2-8048-12f457a7a449"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:18:22.486425 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.486344    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecbaebf9-cb27-42d2-8048-12f457a7a449-maas-api-tls\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:18:22.486425 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.486374    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nxkf\" (UniqueName: \"kubernetes.io/projected/ecbaebf9-cb27-42d2-8048-12f457a7a449-kube-api-access-2nxkf\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:18:22.486425 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.486385    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04dec036-ded6-482a-9193-1d3b8b60e9be-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:18:22.486425 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.486393    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecbaebf9-cb27-42d2-8048-12f457a7a449-metrics-certs\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:18:22.486425 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.486401    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecbaebf9-cb27-42d2-8048-12f457a7a449-openshift-service-ca\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:18:22.486425 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:22.486409    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2hxp\" (UniqueName: \"kubernetes.io/projected/04dec036-ded6-482a-9193-1d3b8b60e9be-kube-api-access-t2hxp\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:18:23.072372 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:23.072328    2577 generic.go:356] "Generic (PLEG): container finished" podID="04dec036-ded6-482a-9193-1d3b8b60e9be" containerID="3a99c57ba018f79386849bdfbfef852d06f0af3b9ce7956224e0278594a6b71a" exitCode=0
Sep 11 16:18:23.072816 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:23.072388    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm"
Sep 11 16:18:23.072816 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:23.072415    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm" event={"ID":"04dec036-ded6-482a-9193-1d3b8b60e9be","Type":"ContainerDied","Data":"3a99c57ba018f79386849bdfbfef852d06f0af3b9ce7956224e0278594a6b71a"}
Sep 11 16:18:23.072816 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:23.072507    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm" event={"ID":"04dec036-ded6-482a-9193-1d3b8b60e9be","Type":"ContainerDied","Data":"5402c44f59280216d1f4aa1d659f4f42c66208b5146e8cf3805d8da61f700fbc"}
Sep 11 16:18:23.072816 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:23.072531    2577 scope.go:117] "RemoveContainer" containerID="3a99c57ba018f79386849bdfbfef852d06f0af3b9ce7956224e0278594a6b71a"
Sep 11 16:18:23.073908 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:23.073893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr" event={"ID":"ecbaebf9-cb27-42d2-8048-12f457a7a449","Type":"ContainerDied","Data":"675ace26d30c91a395ac2edc5f0f98e318664e96d626d91b5492008334be13cf"}
Sep 11 16:18:23.073995 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:23.073944    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"
Sep 11 16:18:23.081846 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:23.081828    2577 scope.go:117] "RemoveContainer" containerID="3a99c57ba018f79386849bdfbfef852d06f0af3b9ce7956224e0278594a6b71a"
Sep 11 16:18:23.082218 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:18:23.082187    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a99c57ba018f79386849bdfbfef852d06f0af3b9ce7956224e0278594a6b71a\": container with ID starting with 3a99c57ba018f79386849bdfbfef852d06f0af3b9ce7956224e0278594a6b71a not found: ID does not exist" containerID="3a99c57ba018f79386849bdfbfef852d06f0af3b9ce7956224e0278594a6b71a"
Sep 11 16:18:23.082298 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:23.082226    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a99c57ba018f79386849bdfbfef852d06f0af3b9ce7956224e0278594a6b71a"} err="failed to get container status \"3a99c57ba018f79386849bdfbfef852d06f0af3b9ce7956224e0278594a6b71a\": rpc error: code = NotFound desc = could not find container \"3a99c57ba018f79386849bdfbfef852d06f0af3b9ce7956224e0278594a6b71a\": container with ID starting with 3a99c57ba018f79386849bdfbfef852d06f0af3b9ce7956224e0278594a6b71a not found: ID does not exist"
Sep 11 16:18:23.082298 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:23.082249    2577 scope.go:117] "RemoveContainer" containerID="f2e9f7aa12750d5d2c08b26d17dc121fe67c801286105c12b447103f778c3d0d"
Sep 11 16:18:23.115262 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:23.115230    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm"]
Sep 11 16:18:23.121269 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:23.121239    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a9b1abbc-5c76bdf46-8fwtm"]
Sep 11 16:18:23.143362 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:23.143329    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"]
Sep 11 16:18:23.159359 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:23.159331    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a9b1abbc-5c79cb67dd-22wpr"]
Sep 11 16:18:24.633361 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:24.633331    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04dec036-ded6-482a-9193-1d3b8b60e9be" path="/var/lib/kubelet/pods/04dec036-ded6-482a-9193-1d3b8b60e9be/volumes"
Sep 11 16:18:24.633729 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:24.633695    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecbaebf9-cb27-42d2-8048-12f457a7a449" path="/var/lib/kubelet/pods/ecbaebf9-cb27-42d2-8048-12f457a7a449/volumes"
Sep 11 16:18:25.621114 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.621082    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"]
Sep 11 16:18:25.621532 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.621519    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecbaebf9-cb27-42d2-8048-12f457a7a449" containerName="maas-api"
Sep 11 16:18:25.621588 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.621534    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecbaebf9-cb27-42d2-8048-12f457a7a449" containerName="maas-api"
Sep 11 16:18:25.621588 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.621552    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04dec036-ded6-482a-9193-1d3b8b60e9be" containerName="payload-pre-processing"
Sep 11 16:18:25.621588 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.621559    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="04dec036-ded6-482a-9193-1d3b8b60e9be" containerName="payload-pre-processing"
Sep 11 16:18:25.621695 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.621635    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecbaebf9-cb27-42d2-8048-12f457a7a449" containerName="maas-api"
Sep 11 16:18:25.621695 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.621642    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="04dec036-ded6-482a-9193-1d3b8b60e9be" containerName="payload-pre-processing"
Sep 11 16:18:25.626336 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.626320    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:25.632876 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.632592    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-99877a\"/\"openshift-service-ca.crt\""
Sep 11 16:18:25.632876 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.632592    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-99877a\"/\"kube-root-ca.crt\""
Sep 11 16:18:25.635138 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.635118    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-99877a\"/\"default-dockercfg-89q2q\""
Sep 11 16:18:25.635412 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.635180    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-99877a\"/\"test-model-99877a-kserve-self-signed-certs\""
Sep 11 16:18:25.652234 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.652208    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"]
Sep 11 16:18:25.715893 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.715861    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-model-cache\") pod \"test-model-99877a-kserve-7fdf9567c5-j5d67\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") " pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:25.716080 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.715984    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-home\") pod \"test-model-99877a-kserve-7fdf9567c5-j5d67\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") " pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:25.716080 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.716056    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-tmp-dir\") pod \"test-model-99877a-kserve-7fdf9567c5-j5d67\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") " pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:25.716184 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.716146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zndm8\" (UniqueName: \"kubernetes.io/projected/9b38ee3f-1329-4007-a7d0-92a17af2f503-kube-api-access-zndm8\") pod \"test-model-99877a-kserve-7fdf9567c5-j5d67\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") " pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:25.716225 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.716193    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-dshm\") pod \"test-model-99877a-kserve-7fdf9567c5-j5d67\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") " pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:25.716225 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.716212    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b38ee3f-1329-4007-a7d0-92a17af2f503-tls-certs\") pod \"test-model-99877a-kserve-7fdf9567c5-j5d67\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") " pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:25.817416 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.817377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-home\") pod \"test-model-99877a-kserve-7fdf9567c5-j5d67\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") " pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:25.817416 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.817414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-tmp-dir\") pod \"test-model-99877a-kserve-7fdf9567c5-j5d67\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") " pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:25.817664 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.817447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zndm8\" (UniqueName: \"kubernetes.io/projected/9b38ee3f-1329-4007-a7d0-92a17af2f503-kube-api-access-zndm8\") pod \"test-model-99877a-kserve-7fdf9567c5-j5d67\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") " pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:25.817664 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.817493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-dshm\") pod \"test-model-99877a-kserve-7fdf9567c5-j5d67\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") " pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:25.817664 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.817518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b38ee3f-1329-4007-a7d0-92a17af2f503-tls-certs\") pod \"test-model-99877a-kserve-7fdf9567c5-j5d67\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") " pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:25.817664 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.817554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-model-cache\") pod \"test-model-99877a-kserve-7fdf9567c5-j5d67\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") " pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:25.817893 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.817848    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-tmp-dir\") pod \"test-model-99877a-kserve-7fdf9567c5-j5d67\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") " pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:25.817893 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.817881    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-model-cache\") pod \"test-model-99877a-kserve-7fdf9567c5-j5d67\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") " pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:25.818153 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.818131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-home\") pod \"test-model-99877a-kserve-7fdf9567c5-j5d67\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") " pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:25.819925 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.819904    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-dshm\") pod \"test-model-99877a-kserve-7fdf9567c5-j5d67\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") " pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:25.820120 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.820103    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b38ee3f-1329-4007-a7d0-92a17af2f503-tls-certs\") pod \"test-model-99877a-kserve-7fdf9567c5-j5d67\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") " pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:25.829176 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.829158    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zndm8\" (UniqueName: \"kubernetes.io/projected/9b38ee3f-1329-4007-a7d0-92a17af2f503-kube-api-access-zndm8\") pod \"test-model-99877a-kserve-7fdf9567c5-j5d67\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") " pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:25.935630 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:25.935551    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:26.228278 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:26.228248    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"]
Sep 11 16:18:26.229866 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:18:26.229834    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b38ee3f_1329_4007_a7d0_92a17af2f503.slice/crio-8ab09c541dd9977c51bf1668c5dac9c45ee4574f96856ebfe13348997923d8b2 WatchSource:0}: Error finding container 8ab09c541dd9977c51bf1668c5dac9c45ee4574f96856ebfe13348997923d8b2: Status 404 returned error can't find the container with id 8ab09c541dd9977c51bf1668c5dac9c45ee4574f96856ebfe13348997923d8b2
Sep 11 16:18:27.097597 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:27.097554    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67" event={"ID":"9b38ee3f-1329-4007-a7d0-92a17af2f503","Type":"ContainerStarted","Data":"1f02a7a443b17654bc89437d6ca87761be27138fd2c552eb0c6528336f969d26"}
Sep 11 16:18:27.097597 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:27.097600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67" event={"ID":"9b38ee3f-1329-4007-a7d0-92a17af2f503","Type":"ContainerStarted","Data":"8ab09c541dd9977c51bf1668c5dac9c45ee4574f96856ebfe13348997923d8b2"}
Sep 11 16:18:27.128874 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:27.128821    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67" podStartSLOduration=2.128804958 podStartE2EDuration="2.128804958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:27.128265241 +0000 UTC m=+966.973908799" watchObservedRunningTime="2026-09-11 16:18:27.128804958 +0000 UTC m=+966.974448512"
Sep 11 16:18:35.935897 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:35.935861    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:35.936301 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:35.935965    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:35.948534 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:35.948509    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:36.155218 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:36.155190    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:18:40.564811 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:40.564771    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"]
Sep 11 16:18:40.565342 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:40.565202    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m" podUID="7d08bd47-2863-41a6-a6bf-23725684c3b3" containerName="maas-api" containerID="cri-o://7a52246c685538fb18b40e41c53ba410154de1683139599b3df9f2c95f6e504d" gracePeriod=30
Sep 11 16:18:40.633658 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:40.633619    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj"]
Sep 11 16:18:40.633880 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:40.633832    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj" podUID="62109c5b-7cf9-4040-b0f3-f7c2e5ba6024" containerName="payload-pre-processing" containerID="cri-o://1eb5cd39cf78c12062a9d9724a62fbfec44f5762a8160075322e10b4a5eb3d01" gracePeriod=30
Sep 11 16:18:40.913260 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:40.913230    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"
Sep 11 16:18:40.917012 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:40.916983    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj"
Sep 11 16:18:40.959367 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:40.959330    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d08bd47-2863-41a6-a6bf-23725684c3b3-maas-api-tls\") pod \"7d08bd47-2863-41a6-a6bf-23725684c3b3\" (UID: \"7d08bd47-2863-41a6-a6bf-23725684c3b3\") "
Sep 11 16:18:40.959367 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:40.959364    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d08bd47-2863-41a6-a6bf-23725684c3b3-metrics-certs\") pod \"7d08bd47-2863-41a6-a6bf-23725684c3b3\" (UID: \"7d08bd47-2863-41a6-a6bf-23725684c3b3\") "
Sep 11 16:18:40.959617 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:40.959407    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xflpt\" (UniqueName: \"kubernetes.io/projected/62109c5b-7cf9-4040-b0f3-f7c2e5ba6024-kube-api-access-xflpt\") pod \"62109c5b-7cf9-4040-b0f3-f7c2e5ba6024\" (UID: \"62109c5b-7cf9-4040-b0f3-f7c2e5ba6024\") "
Sep 11 16:18:40.959617 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:40.959432    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d08bd47-2863-41a6-a6bf-23725684c3b3-openshift-service-ca\") pod \"7d08bd47-2863-41a6-a6bf-23725684c3b3\" (UID: \"7d08bd47-2863-41a6-a6bf-23725684c3b3\") "
Sep 11 16:18:40.959617 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:40.959455    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62109c5b-7cf9-4040-b0f3-f7c2e5ba6024-plugins-config-volume\") pod \"62109c5b-7cf9-4040-b0f3-f7c2e5ba6024\" (UID: \"62109c5b-7cf9-4040-b0f3-f7c2e5ba6024\") "
Sep 11 16:18:40.959827 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:40.959781    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d08bd47-2863-41a6-a6bf-23725684c3b3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7d08bd47-2863-41a6-a6bf-23725684c3b3" (UID: "7d08bd47-2863-41a6-a6bf-23725684c3b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:18:40.959903 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:40.959865    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62109c5b-7cf9-4040-b0f3-f7c2e5ba6024-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "62109c5b-7cf9-4040-b0f3-f7c2e5ba6024" (UID: "62109c5b-7cf9-4040-b0f3-f7c2e5ba6024"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:18:40.961733 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:40.961702    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d08bd47-2863-41a6-a6bf-23725684c3b3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7d08bd47-2863-41a6-a6bf-23725684c3b3" (UID: "7d08bd47-2863-41a6-a6bf-23725684c3b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:18:40.961848 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:40.961767    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d08bd47-2863-41a6-a6bf-23725684c3b3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7d08bd47-2863-41a6-a6bf-23725684c3b3" (UID: "7d08bd47-2863-41a6-a6bf-23725684c3b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:18:40.961848 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:40.961765    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62109c5b-7cf9-4040-b0f3-f7c2e5ba6024-kube-api-access-xflpt" (OuterVolumeSpecName: "kube-api-access-xflpt") pod "62109c5b-7cf9-4040-b0f3-f7c2e5ba6024" (UID: "62109c5b-7cf9-4040-b0f3-f7c2e5ba6024"). InnerVolumeSpecName "kube-api-access-xflpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:18:41.060153 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.060119    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghb99\" (UniqueName: \"kubernetes.io/projected/7d08bd47-2863-41a6-a6bf-23725684c3b3-kube-api-access-ghb99\") pod \"7d08bd47-2863-41a6-a6bf-23725684c3b3\" (UID: \"7d08bd47-2863-41a6-a6bf-23725684c3b3\") "
Sep 11 16:18:41.060371 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.060353    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d08bd47-2863-41a6-a6bf-23725684c3b3-openshift-service-ca\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:18:41.060449 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.060378    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62109c5b-7cf9-4040-b0f3-f7c2e5ba6024-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:18:41.060449 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.060392    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d08bd47-2863-41a6-a6bf-23725684c3b3-maas-api-tls\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:18:41.060449 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.060405    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d08bd47-2863-41a6-a6bf-23725684c3b3-metrics-certs\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:18:41.060449 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.060436    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xflpt\" (UniqueName: \"kubernetes.io/projected/62109c5b-7cf9-4040-b0f3-f7c2e5ba6024-kube-api-access-xflpt\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:18:41.062369 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.062344    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d08bd47-2863-41a6-a6bf-23725684c3b3-kube-api-access-ghb99" (OuterVolumeSpecName: "kube-api-access-ghb99") pod "7d08bd47-2863-41a6-a6bf-23725684c3b3" (UID: "7d08bd47-2863-41a6-a6bf-23725684c3b3"). InnerVolumeSpecName "kube-api-access-ghb99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:18:41.160834 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.160788    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghb99\" (UniqueName: \"kubernetes.io/projected/7d08bd47-2863-41a6-a6bf-23725684c3b3-kube-api-access-ghb99\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:18:41.173421 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.173388    2577 generic.go:356] "Generic (PLEG): container finished" podID="7d08bd47-2863-41a6-a6bf-23725684c3b3" containerID="7a52246c685538fb18b40e41c53ba410154de1683139599b3df9f2c95f6e504d" exitCode=0
Sep 11 16:18:41.173569 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.173435    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"
Sep 11 16:18:41.173569 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.173463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m" event={"ID":"7d08bd47-2863-41a6-a6bf-23725684c3b3","Type":"ContainerDied","Data":"7a52246c685538fb18b40e41c53ba410154de1683139599b3df9f2c95f6e504d"}
Sep 11 16:18:41.173569 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.173520    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m" event={"ID":"7d08bd47-2863-41a6-a6bf-23725684c3b3","Type":"ContainerDied","Data":"2feaaf2b2c8d4c75a0b1e586a47cfb2898c2a9724bf18d3f4da74dde55e92cab"}
Sep 11 16:18:41.173569 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.173551    2577 scope.go:117] "RemoveContainer" containerID="7a52246c685538fb18b40e41c53ba410154de1683139599b3df9f2c95f6e504d"
Sep 11 16:18:41.174795 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.174777    2577 generic.go:356] "Generic (PLEG): container finished" podID="62109c5b-7cf9-4040-b0f3-f7c2e5ba6024" containerID="1eb5cd39cf78c12062a9d9724a62fbfec44f5762a8160075322e10b4a5eb3d01" exitCode=0
Sep 11 16:18:41.174886 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.174824    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj"
Sep 11 16:18:41.174886 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.174844    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj" event={"ID":"62109c5b-7cf9-4040-b0f3-f7c2e5ba6024","Type":"ContainerDied","Data":"1eb5cd39cf78c12062a9d9724a62fbfec44f5762a8160075322e10b4a5eb3d01"}
Sep 11 16:18:41.174886 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.174879    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj" event={"ID":"62109c5b-7cf9-4040-b0f3-f7c2e5ba6024","Type":"ContainerDied","Data":"a4650789ab1a8582d305197de0dab9a18e03b9b18c2c06bfe785b0dfa93ac96d"}
Sep 11 16:18:41.183618 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.183602    2577 scope.go:117] "RemoveContainer" containerID="7a52246c685538fb18b40e41c53ba410154de1683139599b3df9f2c95f6e504d"
Sep 11 16:18:41.183931 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:18:41.183908    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a52246c685538fb18b40e41c53ba410154de1683139599b3df9f2c95f6e504d\": container with ID starting with 7a52246c685538fb18b40e41c53ba410154de1683139599b3df9f2c95f6e504d not found: ID does not exist" containerID="7a52246c685538fb18b40e41c53ba410154de1683139599b3df9f2c95f6e504d"
Sep 11 16:18:41.184015 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.183945    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a52246c685538fb18b40e41c53ba410154de1683139599b3df9f2c95f6e504d"} err="failed to get container status \"7a52246c685538fb18b40e41c53ba410154de1683139599b3df9f2c95f6e504d\": rpc error: code = NotFound desc = could not find container \"7a52246c685538fb18b40e41c53ba410154de1683139599b3df9f2c95f6e504d\": container with ID starting with 7a52246c685538fb18b40e41c53ba410154de1683139599b3df9f2c95f6e504d not found: ID does not exist"
Sep 11 16:18:41.184015 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.183968    2577 scope.go:117] "RemoveContainer" containerID="1eb5cd39cf78c12062a9d9724a62fbfec44f5762a8160075322e10b4a5eb3d01"
Sep 11 16:18:41.192251 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.192235    2577 scope.go:117] "RemoveContainer" containerID="1eb5cd39cf78c12062a9d9724a62fbfec44f5762a8160075322e10b4a5eb3d01"
Sep 11 16:18:41.192505 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:18:41.192489    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb5cd39cf78c12062a9d9724a62fbfec44f5762a8160075322e10b4a5eb3d01\": container with ID starting with 1eb5cd39cf78c12062a9d9724a62fbfec44f5762a8160075322e10b4a5eb3d01 not found: ID does not exist" containerID="1eb5cd39cf78c12062a9d9724a62fbfec44f5762a8160075322e10b4a5eb3d01"
Sep 11 16:18:41.192567 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.192515    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb5cd39cf78c12062a9d9724a62fbfec44f5762a8160075322e10b4a5eb3d01"} err="failed to get container status \"1eb5cd39cf78c12062a9d9724a62fbfec44f5762a8160075322e10b4a5eb3d01\": rpc error: code = NotFound desc = could not find container \"1eb5cd39cf78c12062a9d9724a62fbfec44f5762a8160075322e10b4a5eb3d01\": container with ID starting with 1eb5cd39cf78c12062a9d9724a62fbfec44f5762a8160075322e10b4a5eb3d01 not found: ID does not exist"
Sep 11 16:18:41.208829 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.208792    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"]
Sep 11 16:18:41.214456 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.214430    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-e12be0-647bf5f54b-jfb6m"]
Sep 11 16:18:41.233898 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.233867    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj"]
Sep 11 16:18:41.235546 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:41.235521    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-e12be0-8657f74c4b-58blj"]
Sep 11 16:18:42.632119 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:42.632083    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62109c5b-7cf9-4040-b0f3-f7c2e5ba6024" path="/var/lib/kubelet/pods/62109c5b-7cf9-4040-b0f3-f7c2e5ba6024/volumes"
Sep 11 16:18:42.632477 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:42.632464    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d08bd47-2863-41a6-a6bf-23725684c3b3" path="/var/lib/kubelet/pods/7d08bd47-2863-41a6-a6bf-23725684c3b3/volumes"
Sep 11 16:18:54.928664 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:54.928632    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j"]
Sep 11 16:18:54.929150 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:54.929000    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62109c5b-7cf9-4040-b0f3-f7c2e5ba6024" containerName="payload-pre-processing"
Sep 11 16:18:54.929150 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:54.929010    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="62109c5b-7cf9-4040-b0f3-f7c2e5ba6024" containerName="payload-pre-processing"
Sep 11 16:18:54.929150 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:54.929025    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d08bd47-2863-41a6-a6bf-23725684c3b3" containerName="maas-api"
Sep 11 16:18:54.929150 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:54.929053    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d08bd47-2863-41a6-a6bf-23725684c3b3" containerName="maas-api"
Sep 11 16:18:54.929150 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:54.929113    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d08bd47-2863-41a6-a6bf-23725684c3b3" containerName="maas-api"
Sep 11 16:18:54.929150 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:54.929122    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="62109c5b-7cf9-4040-b0f3-f7c2e5ba6024" containerName="payload-pre-processing"
Sep 11 16:18:54.932384 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:54.932367    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j"
Sep 11 16:18:54.938290 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:54.938262    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-8a7a3a-dockercfg-k928g\""
Sep 11 16:18:54.940509 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:54.940490    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-8a7a3a\""
Sep 11 16:18:54.960525 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:54.960488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1e91bd-8ca7-42ca-915e-973d8d0120fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j\" (UID: \"2d1e91bd-8ca7-42ca-915e-973d8d0120fe\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j"
Sep 11 16:18:54.960675 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:54.960569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khqz7\" (UniqueName: \"kubernetes.io/projected/2d1e91bd-8ca7-42ca-915e-973d8d0120fe-kube-api-access-khqz7\") pod \"payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j\" (UID: \"2d1e91bd-8ca7-42ca-915e-973d8d0120fe\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j"
Sep 11 16:18:54.961730 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:54.961711    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j"]
Sep 11 16:18:55.035482 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:55.035447    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n"]
Sep 11 16:18:55.039092 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:55.039073    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n"
Sep 11 16:18:55.061427 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:55.061384    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34caa264-d324-4e59-ae8d-615ae0e60699-plugins-config-volume\") pod \"payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n\" (UID: \"34caa264-d324-4e59-ae8d-615ae0e60699\") " pod="openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n"
Sep 11 16:18:55.061612 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:55.061464    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1e91bd-8ca7-42ca-915e-973d8d0120fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j\" (UID: \"2d1e91bd-8ca7-42ca-915e-973d8d0120fe\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j"
Sep 11 16:18:55.061612 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:55.061530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khqz7\" (UniqueName: \"kubernetes.io/projected/2d1e91bd-8ca7-42ca-915e-973d8d0120fe-kube-api-access-khqz7\") pod \"payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j\" (UID: \"2d1e91bd-8ca7-42ca-915e-973d8d0120fe\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j"
Sep 11 16:18:55.061612 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:55.061553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msmvs\" (UniqueName: \"kubernetes.io/projected/34caa264-d324-4e59-ae8d-615ae0e60699-kube-api-access-msmvs\") pod \"payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n\" (UID: \"34caa264-d324-4e59-ae8d-615ae0e60699\") " pod="openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n"
Sep 11 16:18:55.062243 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:55.062220    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1e91bd-8ca7-42ca-915e-973d8d0120fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j\" (UID: \"2d1e91bd-8ca7-42ca-915e-973d8d0120fe\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j"
Sep 11 16:18:55.093967 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:55.093927    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n"]
Sep 11 16:18:55.097063 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:55.097017    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khqz7\" (UniqueName: \"kubernetes.io/projected/2d1e91bd-8ca7-42ca-915e-973d8d0120fe-kube-api-access-khqz7\") pod \"payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j\" (UID: \"2d1e91bd-8ca7-42ca-915e-973d8d0120fe\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j"
Sep 11 16:18:55.162497 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:55.162457    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msmvs\" (UniqueName: \"kubernetes.io/projected/34caa264-d324-4e59-ae8d-615ae0e60699-kube-api-access-msmvs\") pod \"payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n\" (UID: \"34caa264-d324-4e59-ae8d-615ae0e60699\") " pod="openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n"
Sep 11 16:18:55.162694 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:55.162581    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34caa264-d324-4e59-ae8d-615ae0e60699-plugins-config-volume\") pod \"payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n\" (UID: \"34caa264-d324-4e59-ae8d-615ae0e60699\") " pod="openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n"
Sep 11 16:18:55.163361 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:55.163336    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34caa264-d324-4e59-ae8d-615ae0e60699-plugins-config-volume\") pod \"payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n\" (UID: \"34caa264-d324-4e59-ae8d-615ae0e60699\") " pod="openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n"
Sep 11 16:18:55.174249 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:55.174217    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msmvs\" (UniqueName: \"kubernetes.io/projected/34caa264-d324-4e59-ae8d-615ae0e60699-kube-api-access-msmvs\") pod \"payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n\" (UID: \"34caa264-d324-4e59-ae8d-615ae0e60699\") " pod="openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n"
Sep 11 16:18:55.243153 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:55.243066    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j"
Sep 11 16:18:55.349280 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:55.349247    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n"
Sep 11 16:18:55.526552 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:55.526529    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j"]
Sep 11 16:18:55.528715 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:18:55.528690    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1e91bd_8ca7_42ca_915e_973d8d0120fe.slice/crio-ed22ec6a46c176e1ff3e7be0156350854670f8ce9b05b929a9670af911782ea5 WatchSource:0}: Error finding container ed22ec6a46c176e1ff3e7be0156350854670f8ce9b05b929a9670af911782ea5: Status 404 returned error can't find the container with id ed22ec6a46c176e1ff3e7be0156350854670f8ce9b05b929a9670af911782ea5
Sep 11 16:18:55.632661 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:55.632628    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n"]
Sep 11 16:18:55.636377 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:18:55.636347    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34caa264_d324_4e59_ae8d_615ae0e60699.slice/crio-1745045e2bfed4360d78463f35ba5ef6e5dcdd8dbf115cbc59eda6cda5c9a63e WatchSource:0}: Error finding container 1745045e2bfed4360d78463f35ba5ef6e5dcdd8dbf115cbc59eda6cda5c9a63e: Status 404 returned error can't find the container with id 1745045e2bfed4360d78463f35ba5ef6e5dcdd8dbf115cbc59eda6cda5c9a63e
Sep 11 16:18:56.260599 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:56.260561    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n" event={"ID":"34caa264-d324-4e59-ae8d-615ae0e60699","Type":"ContainerStarted","Data":"775f147d30d4c4099b65ddc8c9f32a8fc880c4a453d2d43d741c2a15feab0713"}
Sep 11 16:18:56.261076 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:56.260631    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n" event={"ID":"34caa264-d324-4e59-ae8d-615ae0e60699","Type":"ContainerStarted","Data":"1745045e2bfed4360d78463f35ba5ef6e5dcdd8dbf115cbc59eda6cda5c9a63e"}
Sep 11 16:18:56.261076 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:56.260668    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n"
Sep 11 16:18:56.262292 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:56.262269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j" event={"ID":"2d1e91bd-8ca7-42ca-915e-973d8d0120fe","Type":"ContainerStarted","Data":"c5cdf8dafca1d2f0f63dd39eff86531c1d3f51efe02b7dd7767163cb95dec159"}
Sep 11 16:18:56.262425 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:56.262307    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j" event={"ID":"2d1e91bd-8ca7-42ca-915e-973d8d0120fe","Type":"ContainerStarted","Data":"ed22ec6a46c176e1ff3e7be0156350854670f8ce9b05b929a9670af911782ea5"}
Sep 11 16:18:56.262425 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:56.262337    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j"
Sep 11 16:18:56.281875 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:56.281815    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n" podStartSLOduration=2.281797933 podStartE2EDuration="2.281797933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:56.279798555 +0000 UTC m=+996.125442110" watchObservedRunningTime="2026-09-11 16:18:56.281797933 +0000 UTC m=+996.127441478"
Sep 11 16:18:56.302747 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:18:56.302693    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j" podStartSLOduration=2.302673065 podStartE2EDuration="2.302673065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:56.301017965 +0000 UTC m=+996.146661522" watchObservedRunningTime="2026-09-11 16:18:56.302673065 +0000 UTC m=+996.148316670"
Sep 11 16:19:07.271085 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:07.271028    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n"
Sep 11 16:19:07.271521 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:07.271123    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j"
Sep 11 16:19:52.712181 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:52.712143    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n"]
Sep 11 16:19:52.712806 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:52.712400    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n" podUID="34caa264-d324-4e59-ae8d-615ae0e60699" containerName="payload-processing" containerID="cri-o://775f147d30d4c4099b65ddc8c9f32a8fc880c4a453d2d43d741c2a15feab0713" gracePeriod=30
Sep 11 16:19:52.722153 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:52.722122    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j"]
Sep 11 16:19:52.722344 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:52.722324    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j" podUID="2d1e91bd-8ca7-42ca-915e-973d8d0120fe" containerName="payload-pre-processing" containerID="cri-o://c5cdf8dafca1d2f0f63dd39eff86531c1d3f51efe02b7dd7767163cb95dec159" gracePeriod=30
Sep 11 16:19:53.001343 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.001317    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n"
Sep 11 16:19:53.004438 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.004414    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j"
Sep 11 16:19:53.077390 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.077358    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34caa264-d324-4e59-ae8d-615ae0e60699-plugins-config-volume\") pod \"34caa264-d324-4e59-ae8d-615ae0e60699\" (UID: \"34caa264-d324-4e59-ae8d-615ae0e60699\") "
Sep 11 16:19:53.077390 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.077395    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1e91bd-8ca7-42ca-915e-973d8d0120fe-plugins-config-volume\") pod \"2d1e91bd-8ca7-42ca-915e-973d8d0120fe\" (UID: \"2d1e91bd-8ca7-42ca-915e-973d8d0120fe\") "
Sep 11 16:19:53.077654 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.077413    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msmvs\" (UniqueName: \"kubernetes.io/projected/34caa264-d324-4e59-ae8d-615ae0e60699-kube-api-access-msmvs\") pod \"34caa264-d324-4e59-ae8d-615ae0e60699\" (UID: \"34caa264-d324-4e59-ae8d-615ae0e60699\") "
Sep 11 16:19:53.077654 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.077583    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khqz7\" (UniqueName: \"kubernetes.io/projected/2d1e91bd-8ca7-42ca-915e-973d8d0120fe-kube-api-access-khqz7\") pod \"2d1e91bd-8ca7-42ca-915e-973d8d0120fe\" (UID: \"2d1e91bd-8ca7-42ca-915e-973d8d0120fe\") "
Sep 11 16:19:53.077850 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.077797    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1e91bd-8ca7-42ca-915e-973d8d0120fe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d1e91bd-8ca7-42ca-915e-973d8d0120fe" (UID: "2d1e91bd-8ca7-42ca-915e-973d8d0120fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:53.077850 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.077802    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34caa264-d324-4e59-ae8d-615ae0e60699-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "34caa264-d324-4e59-ae8d-615ae0e60699" (UID: "34caa264-d324-4e59-ae8d-615ae0e60699"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:53.078081 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.077984    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34caa264-d324-4e59-ae8d-615ae0e60699-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:19:53.078081 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.078004    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1e91bd-8ca7-42ca-915e-973d8d0120fe-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:19:53.079861 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.079824    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1e91bd-8ca7-42ca-915e-973d8d0120fe-kube-api-access-khqz7" (OuterVolumeSpecName: "kube-api-access-khqz7") pod "2d1e91bd-8ca7-42ca-915e-973d8d0120fe" (UID: "2d1e91bd-8ca7-42ca-915e-973d8d0120fe"). InnerVolumeSpecName "kube-api-access-khqz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:19:53.079985 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.079857    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34caa264-d324-4e59-ae8d-615ae0e60699-kube-api-access-msmvs" (OuterVolumeSpecName: "kube-api-access-msmvs") pod "34caa264-d324-4e59-ae8d-615ae0e60699" (UID: "34caa264-d324-4e59-ae8d-615ae0e60699"). InnerVolumeSpecName "kube-api-access-msmvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:19:53.179138 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.179097    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khqz7\" (UniqueName: \"kubernetes.io/projected/2d1e91bd-8ca7-42ca-915e-973d8d0120fe-kube-api-access-khqz7\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:19:53.179138 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.179128    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msmvs\" (UniqueName: \"kubernetes.io/projected/34caa264-d324-4e59-ae8d-615ae0e60699-kube-api-access-msmvs\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:19:53.561678 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.561636    2577 generic.go:356] "Generic (PLEG): container finished" podID="34caa264-d324-4e59-ae8d-615ae0e60699" containerID="775f147d30d4c4099b65ddc8c9f32a8fc880c4a453d2d43d741c2a15feab0713" exitCode=0
Sep 11 16:19:53.561678 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.561679    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n"
Sep 11 16:19:53.561919 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.561724    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n" event={"ID":"34caa264-d324-4e59-ae8d-615ae0e60699","Type":"ContainerDied","Data":"775f147d30d4c4099b65ddc8c9f32a8fc880c4a453d2d43d741c2a15feab0713"}
Sep 11 16:19:53.561919 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.561774    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n" event={"ID":"34caa264-d324-4e59-ae8d-615ae0e60699","Type":"ContainerDied","Data":"1745045e2bfed4360d78463f35ba5ef6e5dcdd8dbf115cbc59eda6cda5c9a63e"}
Sep 11 16:19:53.561919 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.561801    2577 scope.go:117] "RemoveContainer" containerID="775f147d30d4c4099b65ddc8c9f32a8fc880c4a453d2d43d741c2a15feab0713"
Sep 11 16:19:53.563354 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.563333    2577 generic.go:356] "Generic (PLEG): container finished" podID="2d1e91bd-8ca7-42ca-915e-973d8d0120fe" containerID="c5cdf8dafca1d2f0f63dd39eff86531c1d3f51efe02b7dd7767163cb95dec159" exitCode=0
Sep 11 16:19:53.563475 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.563393    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j"
Sep 11 16:19:53.563475 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.563431    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j" event={"ID":"2d1e91bd-8ca7-42ca-915e-973d8d0120fe","Type":"ContainerDied","Data":"c5cdf8dafca1d2f0f63dd39eff86531c1d3f51efe02b7dd7767163cb95dec159"}
Sep 11 16:19:53.563575 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.563498    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j" event={"ID":"2d1e91bd-8ca7-42ca-915e-973d8d0120fe","Type":"ContainerDied","Data":"ed22ec6a46c176e1ff3e7be0156350854670f8ce9b05b929a9670af911782ea5"}
Sep 11 16:19:53.571248 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.571224    2577 scope.go:117] "RemoveContainer" containerID="775f147d30d4c4099b65ddc8c9f32a8fc880c4a453d2d43d741c2a15feab0713"
Sep 11 16:19:53.571613 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:19:53.571590    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"775f147d30d4c4099b65ddc8c9f32a8fc880c4a453d2d43d741c2a15feab0713\": container with ID starting with 775f147d30d4c4099b65ddc8c9f32a8fc880c4a453d2d43d741c2a15feab0713 not found: ID does not exist" containerID="775f147d30d4c4099b65ddc8c9f32a8fc880c4a453d2d43d741c2a15feab0713"
Sep 11 16:19:53.571725 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.571620    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775f147d30d4c4099b65ddc8c9f32a8fc880c4a453d2d43d741c2a15feab0713"} err="failed to get container status \"775f147d30d4c4099b65ddc8c9f32a8fc880c4a453d2d43d741c2a15feab0713\": rpc error: code = NotFound desc = could not find container \"775f147d30d4c4099b65ddc8c9f32a8fc880c4a453d2d43d741c2a15feab0713\": container with ID starting with 775f147d30d4c4099b65ddc8c9f32a8fc880c4a453d2d43d741c2a15feab0713 not found: ID does not exist"
Sep 11 16:19:53.571725 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.571637    2577 scope.go:117] "RemoveContainer" containerID="c5cdf8dafca1d2f0f63dd39eff86531c1d3f51efe02b7dd7767163cb95dec159"
Sep 11 16:19:53.579997 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.579979    2577 scope.go:117] "RemoveContainer" containerID="c5cdf8dafca1d2f0f63dd39eff86531c1d3f51efe02b7dd7767163cb95dec159"
Sep 11 16:19:53.580349 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:19:53.580331    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5cdf8dafca1d2f0f63dd39eff86531c1d3f51efe02b7dd7767163cb95dec159\": container with ID starting with c5cdf8dafca1d2f0f63dd39eff86531c1d3f51efe02b7dd7767163cb95dec159 not found: ID does not exist" containerID="c5cdf8dafca1d2f0f63dd39eff86531c1d3f51efe02b7dd7767163cb95dec159"
Sep 11 16:19:53.580413 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.580363    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5cdf8dafca1d2f0f63dd39eff86531c1d3f51efe02b7dd7767163cb95dec159"} err="failed to get container status \"c5cdf8dafca1d2f0f63dd39eff86531c1d3f51efe02b7dd7767163cb95dec159\": rpc error: code = NotFound desc = could not find container \"c5cdf8dafca1d2f0f63dd39eff86531c1d3f51efe02b7dd7767163cb95dec159\": container with ID starting with c5cdf8dafca1d2f0f63dd39eff86531c1d3f51efe02b7dd7767163cb95dec159 not found: ID does not exist"
Sep 11 16:19:53.587392 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.587363    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j"]
Sep 11 16:19:53.592149 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.592125    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8a7a3a-84d798ccdb-kj66j"]
Sep 11 16:19:53.603758 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.603726    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n"]
Sep 11 16:19:53.612336 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:53.612298    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8a7a3a-8c74d589-cp64n"]
Sep 11 16:19:54.632912 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:54.632876    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d1e91bd-8ca7-42ca-915e-973d8d0120fe" path="/var/lib/kubelet/pods/2d1e91bd-8ca7-42ca-915e-973d8d0120fe/volumes"
Sep 11 16:19:54.633299 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:19:54.633257    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34caa264-d324-4e59-ae8d-615ae0e60699" path="/var/lib/kubelet/pods/34caa264-d324-4e59-ae8d-615ae0e60699/volumes"
Sep 11 16:20:14.204122 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.204045    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"]
Sep 11 16:20:14.204531 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.204445    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1e91bd-8ca7-42ca-915e-973d8d0120fe" containerName="payload-pre-processing"
Sep 11 16:20:14.204531 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.204458    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1e91bd-8ca7-42ca-915e-973d8d0120fe" containerName="payload-pre-processing"
Sep 11 16:20:14.204531 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.204473    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34caa264-d324-4e59-ae8d-615ae0e60699" containerName="payload-processing"
Sep 11 16:20:14.204531 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.204478    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="34caa264-d324-4e59-ae8d-615ae0e60699" containerName="payload-processing"
Sep 11 16:20:14.204659 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.204541    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="34caa264-d324-4e59-ae8d-615ae0e60699" containerName="payload-processing"
Sep 11 16:20:14.204659 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.204552    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1e91bd-8ca7-42ca-915e-973d8d0120fe" containerName="payload-pre-processing"
Sep 11 16:20:14.209203 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.209182    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"
Sep 11 16:20:14.212981 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.212952    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3882e47c-serving-cert\""
Sep 11 16:20:14.229220 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.229191    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"]
Sep 11 16:20:14.261915 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.261864    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dnm8\" (UniqueName: \"kubernetes.io/projected/d56c75bc-f03a-4444-a433-a9dfef143814-kube-api-access-5dnm8\") pod \"maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv\" (UID: \"d56c75bc-f03a-4444-a433-a9dfef143814\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"
Sep 11 16:20:14.262154 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.261953    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d56c75bc-f03a-4444-a433-a9dfef143814-openshift-service-ca\") pod \"maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv\" (UID: \"d56c75bc-f03a-4444-a433-a9dfef143814\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"
Sep 11 16:20:14.262154 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.261993    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d56c75bc-f03a-4444-a433-a9dfef143814-maas-api-tls\") pod \"maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv\" (UID: \"d56c75bc-f03a-4444-a433-a9dfef143814\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"
Sep 11 16:20:14.262154 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.262069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d56c75bc-f03a-4444-a433-a9dfef143814-metrics-certs\") pod \"maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv\" (UID: \"d56c75bc-f03a-4444-a433-a9dfef143814\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"
Sep 11 16:20:14.363151 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.363116    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d56c75bc-f03a-4444-a433-a9dfef143814-metrics-certs\") pod \"maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv\" (UID: \"d56c75bc-f03a-4444-a433-a9dfef143814\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"
Sep 11 16:20:14.363341 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.363207    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dnm8\" (UniqueName: \"kubernetes.io/projected/d56c75bc-f03a-4444-a433-a9dfef143814-kube-api-access-5dnm8\") pod \"maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv\" (UID: \"d56c75bc-f03a-4444-a433-a9dfef143814\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"
Sep 11 16:20:14.363341 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.363256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d56c75bc-f03a-4444-a433-a9dfef143814-openshift-service-ca\") pod \"maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv\" (UID: \"d56c75bc-f03a-4444-a433-a9dfef143814\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"
Sep 11 16:20:14.363341 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.363283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d56c75bc-f03a-4444-a433-a9dfef143814-maas-api-tls\") pod \"maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv\" (UID: \"d56c75bc-f03a-4444-a433-a9dfef143814\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"
Sep 11 16:20:14.363893 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.363867    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d56c75bc-f03a-4444-a433-a9dfef143814-openshift-service-ca\") pod \"maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv\" (UID: \"d56c75bc-f03a-4444-a433-a9dfef143814\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"
Sep 11 16:20:14.365824 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.365802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d56c75bc-f03a-4444-a433-a9dfef143814-metrics-certs\") pod \"maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv\" (UID: \"d56c75bc-f03a-4444-a433-a9dfef143814\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"
Sep 11 16:20:14.365903 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.365825    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d56c75bc-f03a-4444-a433-a9dfef143814-maas-api-tls\") pod \"maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv\" (UID: \"d56c75bc-f03a-4444-a433-a9dfef143814\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"
Sep 11 16:20:14.378971 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.378947    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dnm8\" (UniqueName: \"kubernetes.io/projected/d56c75bc-f03a-4444-a433-a9dfef143814-kube-api-access-5dnm8\") pod \"maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv\" (UID: \"d56c75bc-f03a-4444-a433-a9dfef143814\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"
Sep 11 16:20:14.520012 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.519917    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"
Sep 11 16:20:14.839824 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.839797    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"]
Sep 11 16:20:14.842469 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:20:14.842429    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56c75bc_f03a_4444_a433_a9dfef143814.slice/crio-adcef9a04cd97982c27deb21e3aa9e1ecec6184a7088a47fe6642bfaea4e3ff4 WatchSource:0}: Error finding container adcef9a04cd97982c27deb21e3aa9e1ecec6184a7088a47fe6642bfaea4e3ff4: Status 404 returned error can't find the container with id adcef9a04cd97982c27deb21e3aa9e1ecec6184a7088a47fe6642bfaea4e3ff4
Sep 11 16:20:14.843750 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:14.843734    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:20:15.392686 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:15.392640    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t"]
Sep 11 16:20:15.396894 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:15.396831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t"
Sep 11 16:20:15.400416 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:15.400387    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3882e47c\""
Sep 11 16:20:15.400837 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:15.400821    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3882e47c-dockercfg-g9mpg\""
Sep 11 16:20:15.473010 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:15.472985    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bztln\" (UniqueName: \"kubernetes.io/projected/f9e8da3f-ec4e-479a-9c52-4725a79b5d2f-kube-api-access-bztln\") pod \"payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t\" (UID: \"f9e8da3f-ec4e-479a-9c52-4725a79b5d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t"
Sep 11 16:20:15.473164 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:15.473019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9e8da3f-ec4e-479a-9c52-4725a79b5d2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t\" (UID: \"f9e8da3f-ec4e-479a-9c52-4725a79b5d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t"
Sep 11 16:20:15.480881 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:15.480855    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t"]
Sep 11 16:20:15.574141 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:15.574106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bztln\" (UniqueName: \"kubernetes.io/projected/f9e8da3f-ec4e-479a-9c52-4725a79b5d2f-kube-api-access-bztln\") pod \"payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t\" (UID: \"f9e8da3f-ec4e-479a-9c52-4725a79b5d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t"
Sep 11 16:20:15.574343 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:15.574152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9e8da3f-ec4e-479a-9c52-4725a79b5d2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t\" (UID: \"f9e8da3f-ec4e-479a-9c52-4725a79b5d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t"
Sep 11 16:20:15.574878 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:15.574859    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9e8da3f-ec4e-479a-9c52-4725a79b5d2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t\" (UID: \"f9e8da3f-ec4e-479a-9c52-4725a79b5d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t"
Sep 11 16:20:15.586399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:15.586365    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bztln\" (UniqueName: \"kubernetes.io/projected/f9e8da3f-ec4e-479a-9c52-4725a79b5d2f-kube-api-access-bztln\") pod \"payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t\" (UID: \"f9e8da3f-ec4e-479a-9c52-4725a79b5d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t"
Sep 11 16:20:15.684073 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:15.683980    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv" event={"ID":"d56c75bc-f03a-4444-a433-a9dfef143814","Type":"ContainerStarted","Data":"bbae4b555a4ca237f730e85fbed32314eacb810a5d774d4735d2afc56e9f2aaa"}
Sep 11 16:20:15.684073 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:15.684028    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv" event={"ID":"d56c75bc-f03a-4444-a433-a9dfef143814","Type":"ContainerStarted","Data":"adcef9a04cd97982c27deb21e3aa9e1ecec6184a7088a47fe6642bfaea4e3ff4"}
Sep 11 16:20:15.684257 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:15.684094    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"
Sep 11 16:20:15.709008 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:15.708983    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t"
Sep 11 16:20:15.736777 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:15.736726    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv" podStartSLOduration=1.158418758 podStartE2EDuration="1.736708042s" totalImagesPullingTime="578.289284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:20:15.732872906 +0000 UTC m=+1075.578516462" watchObservedRunningTime="2026-09-11 16:20:15.736708042 +0000 UTC m=+1075.582351622"
Sep 11 16:20:16.017512 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:16.017486    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t"]
Sep 11 16:20:16.020241 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:20:16.020209    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e8da3f_ec4e_479a_9c52_4725a79b5d2f.slice/crio-8ac3774c8b6a7bf6cadfa1be34080d883e372fc2df99804bb464fed57a5c9ca4 WatchSource:0}: Error finding container 8ac3774c8b6a7bf6cadfa1be34080d883e372fc2df99804bb464fed57a5c9ca4: Status 404 returned error can't find the container with id 8ac3774c8b6a7bf6cadfa1be34080d883e372fc2df99804bb464fed57a5c9ca4
Sep 11 16:20:16.690000 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:16.689965    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t" event={"ID":"f9e8da3f-ec4e-479a-9c52-4725a79b5d2f","Type":"ContainerStarted","Data":"ae65b930acc79cca038e88b872cc0f50a251ec73ebf9004337d1d4b7cc65d4ca"}
Sep 11 16:20:16.690403 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:16.690023    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t" event={"ID":"f9e8da3f-ec4e-479a-9c52-4725a79b5d2f","Type":"ContainerStarted","Data":"8ac3774c8b6a7bf6cadfa1be34080d883e372fc2df99804bb464fed57a5c9ca4"}
Sep 11 16:20:16.732315 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:16.732258    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t" podStartSLOduration=1.732240156 podStartE2EDuration="1.732240156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:20:16.724596455 +0000 UTC m=+1076.570240011" watchObservedRunningTime="2026-09-11 16:20:16.732240156 +0000 UTC m=+1076.577883712"
Sep 11 16:20:17.697755 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:17.697720    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t"
Sep 11 16:20:17.743012 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:17.742968    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"]
Sep 11 16:20:21.695418 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:21.695392    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"
Sep 11 16:20:28.704737 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:28.704706    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t"
Sep 11 16:20:32.762824 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:32.762780    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67" podUID="9b38ee3f-1329-4007-a7d0-92a17af2f503" containerName="main" containerID="cri-o://1f02a7a443b17654bc89437d6ca87761be27138fd2c552eb0c6528336f969d26" gracePeriod=45
Sep 11 16:20:33.022514 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.022449    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:20:33.140129 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.140092    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-tmp-dir\") pod \"9b38ee3f-1329-4007-a7d0-92a17af2f503\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") "
Sep 11 16:20:33.140129 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.140134    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-dshm\") pod \"9b38ee3f-1329-4007-a7d0-92a17af2f503\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") "
Sep 11 16:20:33.140336 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.140208    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b38ee3f-1329-4007-a7d0-92a17af2f503-tls-certs\") pod \"9b38ee3f-1329-4007-a7d0-92a17af2f503\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") "
Sep 11 16:20:33.140336 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.140227    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zndm8\" (UniqueName: \"kubernetes.io/projected/9b38ee3f-1329-4007-a7d0-92a17af2f503-kube-api-access-zndm8\") pod \"9b38ee3f-1329-4007-a7d0-92a17af2f503\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") "
Sep 11 16:20:33.140336 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.140253    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-home\") pod \"9b38ee3f-1329-4007-a7d0-92a17af2f503\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") "
Sep 11 16:20:33.140336 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.140288    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-model-cache\") pod \"9b38ee3f-1329-4007-a7d0-92a17af2f503\" (UID: \"9b38ee3f-1329-4007-a7d0-92a17af2f503\") "
Sep 11 16:20:33.140548 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.140398    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b38ee3f-1329-4007-a7d0-92a17af2f503" (UID: "9b38ee3f-1329-4007-a7d0-92a17af2f503"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:20:33.140608 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.140591    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-tmp-dir\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:20:33.140662 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.140599    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b38ee3f-1329-4007-a7d0-92a17af2f503" (UID: "9b38ee3f-1329-4007-a7d0-92a17af2f503"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:20:33.140662 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.140615    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-home" (OuterVolumeSpecName: "home") pod "9b38ee3f-1329-4007-a7d0-92a17af2f503" (UID: "9b38ee3f-1329-4007-a7d0-92a17af2f503"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:20:33.142778 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.142749    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-dshm" (OuterVolumeSpecName: "dshm") pod "9b38ee3f-1329-4007-a7d0-92a17af2f503" (UID: "9b38ee3f-1329-4007-a7d0-92a17af2f503"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:20:33.142778 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.142757    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b38ee3f-1329-4007-a7d0-92a17af2f503-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b38ee3f-1329-4007-a7d0-92a17af2f503" (UID: "9b38ee3f-1329-4007-a7d0-92a17af2f503"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:33.143017 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.142833    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b38ee3f-1329-4007-a7d0-92a17af2f503-kube-api-access-zndm8" (OuterVolumeSpecName: "kube-api-access-zndm8") pod "9b38ee3f-1329-4007-a7d0-92a17af2f503" (UID: "9b38ee3f-1329-4007-a7d0-92a17af2f503"). InnerVolumeSpecName "kube-api-access-zndm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:20:33.241279 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.241247    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b38ee3f-1329-4007-a7d0-92a17af2f503-tls-certs\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:20:33.241279 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.241275    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zndm8\" (UniqueName: \"kubernetes.io/projected/9b38ee3f-1329-4007-a7d0-92a17af2f503-kube-api-access-zndm8\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:20:33.241279 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.241286    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-home\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:20:33.241508 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.241295    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-model-cache\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:20:33.241508 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.241304    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b38ee3f-1329-4007-a7d0-92a17af2f503-dshm\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:20:33.780215 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.780180    2577 generic.go:356] "Generic (PLEG): container finished" podID="9b38ee3f-1329-4007-a7d0-92a17af2f503" containerID="1f02a7a443b17654bc89437d6ca87761be27138fd2c552eb0c6528336f969d26" exitCode=0
Sep 11 16:20:33.780595 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.780242    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"
Sep 11 16:20:33.780595 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.780258    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67" event={"ID":"9b38ee3f-1329-4007-a7d0-92a17af2f503","Type":"ContainerDied","Data":"1f02a7a443b17654bc89437d6ca87761be27138fd2c552eb0c6528336f969d26"}
Sep 11 16:20:33.780595 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.780310    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67" event={"ID":"9b38ee3f-1329-4007-a7d0-92a17af2f503","Type":"ContainerDied","Data":"8ab09c541dd9977c51bf1668c5dac9c45ee4574f96856ebfe13348997923d8b2"}
Sep 11 16:20:33.780595 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.780334    2577 scope.go:117] "RemoveContainer" containerID="1f02a7a443b17654bc89437d6ca87761be27138fd2c552eb0c6528336f969d26"
Sep 11 16:20:33.790019 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.789998    2577 scope.go:117] "RemoveContainer" containerID="1f02a7a443b17654bc89437d6ca87761be27138fd2c552eb0c6528336f969d26"
Sep 11 16:20:33.790406 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:20:33.790379    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f02a7a443b17654bc89437d6ca87761be27138fd2c552eb0c6528336f969d26\": container with ID starting with 1f02a7a443b17654bc89437d6ca87761be27138fd2c552eb0c6528336f969d26 not found: ID does not exist" containerID="1f02a7a443b17654bc89437d6ca87761be27138fd2c552eb0c6528336f969d26"
Sep 11 16:20:33.790484 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.790421    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f02a7a443b17654bc89437d6ca87761be27138fd2c552eb0c6528336f969d26"} err="failed to get container status \"1f02a7a443b17654bc89437d6ca87761be27138fd2c552eb0c6528336f969d26\": rpc error: code = NotFound desc = could not find container \"1f02a7a443b17654bc89437d6ca87761be27138fd2c552eb0c6528336f969d26\": container with ID starting with 1f02a7a443b17654bc89437d6ca87761be27138fd2c552eb0c6528336f969d26 not found: ID does not exist"
Sep 11 16:20:33.802792 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.802764    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"]
Sep 11 16:20:33.807172 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:33.807145    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-99877a/test-model-99877a-kserve-7fdf9567c5-j5d67"]
Sep 11 16:20:34.636255 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:34.636223    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b38ee3f-1329-4007-a7d0-92a17af2f503" path="/var/lib/kubelet/pods/9b38ee3f-1329-4007-a7d0-92a17af2f503/volumes"
Sep 11 16:20:34.732916 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:34.732884    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"]
Sep 11 16:20:34.733358 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:34.733114    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss" podUID="f70fa3e6-9860-4a19-9e87-24af505c8e94" containerName="maas-api" containerID="cri-o://96936adb087181642bd9f475e5b63b97b5818da936dadab74bf67faa75b8f930" gracePeriod=30
Sep 11 16:20:34.987852 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:34.987824    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"
Sep 11 16:20:35.154902 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.154813    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70fa3e6-9860-4a19-9e87-24af505c8e94-maas-api-tls\") pod \"f70fa3e6-9860-4a19-9e87-24af505c8e94\" (UID: \"f70fa3e6-9860-4a19-9e87-24af505c8e94\") "
Sep 11 16:20:35.154902 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.154853    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70fa3e6-9860-4a19-9e87-24af505c8e94-openshift-service-ca\") pod \"f70fa3e6-9860-4a19-9e87-24af505c8e94\" (UID: \"f70fa3e6-9860-4a19-9e87-24af505c8e94\") "
Sep 11 16:20:35.154902 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.154871    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h966\" (UniqueName: \"kubernetes.io/projected/f70fa3e6-9860-4a19-9e87-24af505c8e94-kube-api-access-6h966\") pod \"f70fa3e6-9860-4a19-9e87-24af505c8e94\" (UID: \"f70fa3e6-9860-4a19-9e87-24af505c8e94\") "
Sep 11 16:20:35.155242 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.154910    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f70fa3e6-9860-4a19-9e87-24af505c8e94-metrics-certs\") pod \"f70fa3e6-9860-4a19-9e87-24af505c8e94\" (UID: \"f70fa3e6-9860-4a19-9e87-24af505c8e94\") "
Sep 11 16:20:35.155319 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.155294    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70fa3e6-9860-4a19-9e87-24af505c8e94-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f70fa3e6-9860-4a19-9e87-24af505c8e94" (UID: "f70fa3e6-9860-4a19-9e87-24af505c8e94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:35.157179 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.157152    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70fa3e6-9860-4a19-9e87-24af505c8e94-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f70fa3e6-9860-4a19-9e87-24af505c8e94" (UID: "f70fa3e6-9860-4a19-9e87-24af505c8e94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:35.157271 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.157185    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70fa3e6-9860-4a19-9e87-24af505c8e94-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f70fa3e6-9860-4a19-9e87-24af505c8e94" (UID: "f70fa3e6-9860-4a19-9e87-24af505c8e94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:35.157271 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.157199    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70fa3e6-9860-4a19-9e87-24af505c8e94-kube-api-access-6h966" (OuterVolumeSpecName: "kube-api-access-6h966") pod "f70fa3e6-9860-4a19-9e87-24af505c8e94" (UID: "f70fa3e6-9860-4a19-9e87-24af505c8e94"). InnerVolumeSpecName "kube-api-access-6h966". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:20:35.255898 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.255859    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f70fa3e6-9860-4a19-9e87-24af505c8e94-maas-api-tls\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:20:35.255898 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.255892    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70fa3e6-9860-4a19-9e87-24af505c8e94-openshift-service-ca\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:20:35.255898 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.255906    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h966\" (UniqueName: \"kubernetes.io/projected/f70fa3e6-9860-4a19-9e87-24af505c8e94-kube-api-access-6h966\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:20:35.256184 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.255918    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f70fa3e6-9860-4a19-9e87-24af505c8e94-metrics-certs\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:20:35.797898 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.797861    2577 generic.go:356] "Generic (PLEG): container finished" podID="f70fa3e6-9860-4a19-9e87-24af505c8e94" containerID="96936adb087181642bd9f475e5b63b97b5818da936dadab74bf67faa75b8f930" exitCode=0
Sep 11 16:20:35.797898 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.797904    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"
Sep 11 16:20:35.798168 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.797940    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss" event={"ID":"f70fa3e6-9860-4a19-9e87-24af505c8e94","Type":"ContainerDied","Data":"96936adb087181642bd9f475e5b63b97b5818da936dadab74bf67faa75b8f930"}
Sep 11 16:20:35.798168 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.797983    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss" event={"ID":"f70fa3e6-9860-4a19-9e87-24af505c8e94","Type":"ContainerDied","Data":"383cd8d85c34755f814a5f3e0525867aab02ebc8129770ad5315424dc6456b43"}
Sep 11 16:20:35.798168 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.798004    2577 scope.go:117] "RemoveContainer" containerID="96936adb087181642bd9f475e5b63b97b5818da936dadab74bf67faa75b8f930"
Sep 11 16:20:35.807852 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.807830    2577 scope.go:117] "RemoveContainer" containerID="96936adb087181642bd9f475e5b63b97b5818da936dadab74bf67faa75b8f930"
Sep 11 16:20:35.808243 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:20:35.808218    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96936adb087181642bd9f475e5b63b97b5818da936dadab74bf67faa75b8f930\": container with ID starting with 96936adb087181642bd9f475e5b63b97b5818da936dadab74bf67faa75b8f930 not found: ID does not exist" containerID="96936adb087181642bd9f475e5b63b97b5818da936dadab74bf67faa75b8f930"
Sep 11 16:20:35.808311 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.808258    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96936adb087181642bd9f475e5b63b97b5818da936dadab74bf67faa75b8f930"} err="failed to get container status \"96936adb087181642bd9f475e5b63b97b5818da936dadab74bf67faa75b8f930\": rpc error: code = NotFound desc = could not find container \"96936adb087181642bd9f475e5b63b97b5818da936dadab74bf67faa75b8f930\": container with ID starting with 96936adb087181642bd9f475e5b63b97b5818da936dadab74bf67faa75b8f930 not found: ID does not exist"
Sep 11 16:20:35.820737 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.820708    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"]
Sep 11 16:20:35.824716 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:35.824687    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-99877a-555cfc9db8-sz6ss"]
Sep 11 16:20:36.632996 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:36.632960    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f70fa3e6-9860-4a19-9e87-24af505c8e94" path="/var/lib/kubelet/pods/f70fa3e6-9860-4a19-9e87-24af505c8e94/volumes"
Sep 11 16:20:53.147650 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.147611    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n"]
Sep 11 16:20:53.148261 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.147904    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n" podUID="d6267331-2bd6-479b-957a-656400a77b57" containerName="payload-pre-processing" containerID="cri-o://6bdf236d3b57adc56817bc96a1ec12d676bb581b7099346260730a3fc57c6a0a" gracePeriod=30
Sep 11 16:20:53.273378 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.273346    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"]
Sep 11 16:20:53.273582 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.273560    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv" podUID="d56c75bc-f03a-4444-a433-a9dfef143814" containerName="maas-api" containerID="cri-o://bbae4b555a4ca237f730e85fbed32314eacb810a5d774d4735d2afc56e9f2aaa" gracePeriod=30
Sep 11 16:20:53.353727 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.353692    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t"]
Sep 11 16:20:53.353950 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.353925    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t" podUID="f9e8da3f-ec4e-479a-9c52-4725a79b5d2f" containerName="payload-pre-processing" containerID="cri-o://ae65b930acc79cca038e88b872cc0f50a251ec73ebf9004337d1d4b7cc65d4ca" gracePeriod=30
Sep 11 16:20:53.475163 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.475130    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n"
Sep 11 16:20:53.512535 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.512468    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hsq8\" (UniqueName: \"kubernetes.io/projected/d6267331-2bd6-479b-957a-656400a77b57-kube-api-access-6hsq8\") pod \"d6267331-2bd6-479b-957a-656400a77b57\" (UID: \"d6267331-2bd6-479b-957a-656400a77b57\") "
Sep 11 16:20:53.512730 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.512542    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6267331-2bd6-479b-957a-656400a77b57-plugins-config-volume\") pod \"d6267331-2bd6-479b-957a-656400a77b57\" (UID: \"d6267331-2bd6-479b-957a-656400a77b57\") "
Sep 11 16:20:53.513197 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.513156    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6267331-2bd6-479b-957a-656400a77b57-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d6267331-2bd6-479b-957a-656400a77b57" (UID: "d6267331-2bd6-479b-957a-656400a77b57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:53.515346 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.515309    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6267331-2bd6-479b-957a-656400a77b57-kube-api-access-6hsq8" (OuterVolumeSpecName: "kube-api-access-6hsq8") pod "d6267331-2bd6-479b-957a-656400a77b57" (UID: "d6267331-2bd6-479b-957a-656400a77b57"). InnerVolumeSpecName "kube-api-access-6hsq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:20:53.614254 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.614227    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6267331-2bd6-479b-957a-656400a77b57-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:20:53.614254 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.614254    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hsq8\" (UniqueName: \"kubernetes.io/projected/d6267331-2bd6-479b-957a-656400a77b57-kube-api-access-6hsq8\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:20:53.619919 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.619893    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"
Sep 11 16:20:53.631080 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.631053    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t"
Sep 11 16:20:53.715395 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.715308    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d56c75bc-f03a-4444-a433-a9dfef143814-openshift-service-ca\") pod \"d56c75bc-f03a-4444-a433-a9dfef143814\" (UID: \"d56c75bc-f03a-4444-a433-a9dfef143814\") "
Sep 11 16:20:53.715395 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.715355    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9e8da3f-ec4e-479a-9c52-4725a79b5d2f-plugins-config-volume\") pod \"f9e8da3f-ec4e-479a-9c52-4725a79b5d2f\" (UID: \"f9e8da3f-ec4e-479a-9c52-4725a79b5d2f\") "
Sep 11 16:20:53.715395 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.715379    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bztln\" (UniqueName: \"kubernetes.io/projected/f9e8da3f-ec4e-479a-9c52-4725a79b5d2f-kube-api-access-bztln\") pod \"f9e8da3f-ec4e-479a-9c52-4725a79b5d2f\" (UID: \"f9e8da3f-ec4e-479a-9c52-4725a79b5d2f\") "
Sep 11 16:20:53.715689 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.715433    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dnm8\" (UniqueName: \"kubernetes.io/projected/d56c75bc-f03a-4444-a433-a9dfef143814-kube-api-access-5dnm8\") pod \"d56c75bc-f03a-4444-a433-a9dfef143814\" (UID: \"d56c75bc-f03a-4444-a433-a9dfef143814\") "
Sep 11 16:20:53.715689 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.715480    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d56c75bc-f03a-4444-a433-a9dfef143814-metrics-certs\") pod \"d56c75bc-f03a-4444-a433-a9dfef143814\" (UID: \"d56c75bc-f03a-4444-a433-a9dfef143814\") "
Sep 11 16:20:53.715689 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.715602    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d56c75bc-f03a-4444-a433-a9dfef143814-maas-api-tls\") pod \"d56c75bc-f03a-4444-a433-a9dfef143814\" (UID: \"d56c75bc-f03a-4444-a433-a9dfef143814\") "
Sep 11 16:20:53.715855 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.715769    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d56c75bc-f03a-4444-a433-a9dfef143814-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d56c75bc-f03a-4444-a433-a9dfef143814" (UID: "d56c75bc-f03a-4444-a433-a9dfef143814"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:53.715855 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.715791    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9e8da3f-ec4e-479a-9c52-4725a79b5d2f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9e8da3f-ec4e-479a-9c52-4725a79b5d2f" (UID: "f9e8da3f-ec4e-479a-9c52-4725a79b5d2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:53.716027 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.716004    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d56c75bc-f03a-4444-a433-a9dfef143814-openshift-service-ca\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:20:53.716438 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.716416    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9e8da3f-ec4e-479a-9c52-4725a79b5d2f-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:20:53.717878 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.717852    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d56c75bc-f03a-4444-a433-a9dfef143814-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d56c75bc-f03a-4444-a433-a9dfef143814" (UID: "d56c75bc-f03a-4444-a433-a9dfef143814"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:53.717978 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.717916    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e8da3f-ec4e-479a-9c52-4725a79b5d2f-kube-api-access-bztln" (OuterVolumeSpecName: "kube-api-access-bztln") pod "f9e8da3f-ec4e-479a-9c52-4725a79b5d2f" (UID: "f9e8da3f-ec4e-479a-9c52-4725a79b5d2f"). InnerVolumeSpecName "kube-api-access-bztln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:20:53.717978 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.717916    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56c75bc-f03a-4444-a433-a9dfef143814-kube-api-access-5dnm8" (OuterVolumeSpecName: "kube-api-access-5dnm8") pod "d56c75bc-f03a-4444-a433-a9dfef143814" (UID: "d56c75bc-f03a-4444-a433-a9dfef143814"). InnerVolumeSpecName "kube-api-access-5dnm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:20:53.718228 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.718213    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d56c75bc-f03a-4444-a433-a9dfef143814-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d56c75bc-f03a-4444-a433-a9dfef143814" (UID: "d56c75bc-f03a-4444-a433-a9dfef143814"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:53.817102 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.817030    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dnm8\" (UniqueName: \"kubernetes.io/projected/d56c75bc-f03a-4444-a433-a9dfef143814-kube-api-access-5dnm8\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:20:53.817102 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.817100    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d56c75bc-f03a-4444-a433-a9dfef143814-metrics-certs\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:20:53.817302 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.817117    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d56c75bc-f03a-4444-a433-a9dfef143814-maas-api-tls\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:20:53.817302 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.817130    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bztln\" (UniqueName: \"kubernetes.io/projected/f9e8da3f-ec4e-479a-9c52-4725a79b5d2f-kube-api-access-bztln\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:20:53.892857 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.892825    2577 generic.go:356] "Generic (PLEG): container finished" podID="d56c75bc-f03a-4444-a433-a9dfef143814" containerID="bbae4b555a4ca237f730e85fbed32314eacb810a5d774d4735d2afc56e9f2aaa" exitCode=0
Sep 11 16:20:53.893082 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.892870    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"
Sep 11 16:20:53.893082 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.892917    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv" event={"ID":"d56c75bc-f03a-4444-a433-a9dfef143814","Type":"ContainerDied","Data":"bbae4b555a4ca237f730e85fbed32314eacb810a5d774d4735d2afc56e9f2aaa"}
Sep 11 16:20:53.893082 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.892974    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv" event={"ID":"d56c75bc-f03a-4444-a433-a9dfef143814","Type":"ContainerDied","Data":"adcef9a04cd97982c27deb21e3aa9e1ecec6184a7088a47fe6642bfaea4e3ff4"}
Sep 11 16:20:53.893082 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.893001    2577 scope.go:117] "RemoveContainer" containerID="bbae4b555a4ca237f730e85fbed32314eacb810a5d774d4735d2afc56e9f2aaa"
Sep 11 16:20:53.894252 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.894230    2577 generic.go:356] "Generic (PLEG): container finished" podID="f9e8da3f-ec4e-479a-9c52-4725a79b5d2f" containerID="ae65b930acc79cca038e88b872cc0f50a251ec73ebf9004337d1d4b7cc65d4ca" exitCode=0
Sep 11 16:20:53.894364 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.894262    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t"
Sep 11 16:20:53.894364 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.894307    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t" event={"ID":"f9e8da3f-ec4e-479a-9c52-4725a79b5d2f","Type":"ContainerDied","Data":"ae65b930acc79cca038e88b872cc0f50a251ec73ebf9004337d1d4b7cc65d4ca"}
Sep 11 16:20:53.894364 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.894360    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t" event={"ID":"f9e8da3f-ec4e-479a-9c52-4725a79b5d2f","Type":"ContainerDied","Data":"8ac3774c8b6a7bf6cadfa1be34080d883e372fc2df99804bb464fed57a5c9ca4"}
Sep 11 16:20:53.895671 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.895648    2577 generic.go:356] "Generic (PLEG): container finished" podID="d6267331-2bd6-479b-957a-656400a77b57" containerID="6bdf236d3b57adc56817bc96a1ec12d676bb581b7099346260730a3fc57c6a0a" exitCode=0
Sep 11 16:20:53.895671 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.895693    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n"
Sep 11 16:20:53.895868 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.895712    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n" event={"ID":"d6267331-2bd6-479b-957a-656400a77b57","Type":"ContainerDied","Data":"6bdf236d3b57adc56817bc96a1ec12d676bb581b7099346260730a3fc57c6a0a"}
Sep 11 16:20:53.895868 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.895747    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n" event={"ID":"d6267331-2bd6-479b-957a-656400a77b57","Type":"ContainerDied","Data":"cfcd6c8614a144d2741250bb9e31e0306e4d2b572d9c7db6dd0159c8446822c0"}
Sep 11 16:20:53.906428 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.906395    2577 scope.go:117] "RemoveContainer" containerID="bbae4b555a4ca237f730e85fbed32314eacb810a5d774d4735d2afc56e9f2aaa"
Sep 11 16:20:53.911267 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:20:53.909433    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbae4b555a4ca237f730e85fbed32314eacb810a5d774d4735d2afc56e9f2aaa\": container with ID starting with bbae4b555a4ca237f730e85fbed32314eacb810a5d774d4735d2afc56e9f2aaa not found: ID does not exist" containerID="bbae4b555a4ca237f730e85fbed32314eacb810a5d774d4735d2afc56e9f2aaa"
Sep 11 16:20:53.911267 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.909478    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbae4b555a4ca237f730e85fbed32314eacb810a5d774d4735d2afc56e9f2aaa"} err="failed to get container status \"bbae4b555a4ca237f730e85fbed32314eacb810a5d774d4735d2afc56e9f2aaa\": rpc error: code = NotFound desc = could not find container \"bbae4b555a4ca237f730e85fbed32314eacb810a5d774d4735d2afc56e9f2aaa\": container with ID starting with bbae4b555a4ca237f730e85fbed32314eacb810a5d774d4735d2afc56e9f2aaa not found: ID does not exist"
Sep 11 16:20:53.911267 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.909503    2577 scope.go:117] "RemoveContainer" containerID="ae65b930acc79cca038e88b872cc0f50a251ec73ebf9004337d1d4b7cc65d4ca"
Sep 11 16:20:53.922127 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.922103    2577 scope.go:117] "RemoveContainer" containerID="ae65b930acc79cca038e88b872cc0f50a251ec73ebf9004337d1d4b7cc65d4ca"
Sep 11 16:20:53.922468 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:20:53.922439    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae65b930acc79cca038e88b872cc0f50a251ec73ebf9004337d1d4b7cc65d4ca\": container with ID starting with ae65b930acc79cca038e88b872cc0f50a251ec73ebf9004337d1d4b7cc65d4ca not found: ID does not exist" containerID="ae65b930acc79cca038e88b872cc0f50a251ec73ebf9004337d1d4b7cc65d4ca"
Sep 11 16:20:53.922588 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.922475    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae65b930acc79cca038e88b872cc0f50a251ec73ebf9004337d1d4b7cc65d4ca"} err="failed to get container status \"ae65b930acc79cca038e88b872cc0f50a251ec73ebf9004337d1d4b7cc65d4ca\": rpc error: code = NotFound desc = could not find container \"ae65b930acc79cca038e88b872cc0f50a251ec73ebf9004337d1d4b7cc65d4ca\": container with ID starting with ae65b930acc79cca038e88b872cc0f50a251ec73ebf9004337d1d4b7cc65d4ca not found: ID does not exist"
Sep 11 16:20:53.922588 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.922492    2577 scope.go:117] "RemoveContainer" containerID="6bdf236d3b57adc56817bc96a1ec12d676bb581b7099346260730a3fc57c6a0a"
Sep 11 16:20:53.930588 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.930569    2577 scope.go:117] "RemoveContainer" containerID="6bdf236d3b57adc56817bc96a1ec12d676bb581b7099346260730a3fc57c6a0a"
Sep 11 16:20:53.930888 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:20:53.930860    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bdf236d3b57adc56817bc96a1ec12d676bb581b7099346260730a3fc57c6a0a\": container with ID starting with 6bdf236d3b57adc56817bc96a1ec12d676bb581b7099346260730a3fc57c6a0a not found: ID does not exist" containerID="6bdf236d3b57adc56817bc96a1ec12d676bb581b7099346260730a3fc57c6a0a"
Sep 11 16:20:53.930968 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.930902    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bdf236d3b57adc56817bc96a1ec12d676bb581b7099346260730a3fc57c6a0a"} err="failed to get container status \"6bdf236d3b57adc56817bc96a1ec12d676bb581b7099346260730a3fc57c6a0a\": rpc error: code = NotFound desc = could not find container \"6bdf236d3b57adc56817bc96a1ec12d676bb581b7099346260730a3fc57c6a0a\": container with ID starting with 6bdf236d3b57adc56817bc96a1ec12d676bb581b7099346260730a3fc57c6a0a not found: ID does not exist"
Sep 11 16:20:53.936461 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.936436    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t"]
Sep 11 16:20:53.943862 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.943839    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3882e47c-86855b97fd-wpc4t"]
Sep 11 16:20:53.961288 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.961259    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n"]
Sep 11 16:20:53.972761 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.972696    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b6b139-7bfd9847cd-kwm6n"]
Sep 11 16:20:53.990190 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.990160    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"]
Sep 11 16:20:53.992371 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:53.992346    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3882e47c-84768f5f6f-7m6mv"]
Sep 11 16:20:54.633646 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:54.633608    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d56c75bc-f03a-4444-a433-a9dfef143814" path="/var/lib/kubelet/pods/d56c75bc-f03a-4444-a433-a9dfef143814/volumes"
Sep 11 16:20:54.634147 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:54.634057    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6267331-2bd6-479b-957a-656400a77b57" path="/var/lib/kubelet/pods/d6267331-2bd6-479b-957a-656400a77b57/volumes"
Sep 11 16:20:54.634433 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:20:54.634414    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9e8da3f-ec4e-479a-9c52-4725a79b5d2f" path="/var/lib/kubelet/pods/f9e8da3f-ec4e-479a-9c52-4725a79b5d2f/volumes"
Sep 11 16:21:07.763540 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.763509    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9"]
Sep 11 16:21:07.763914 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.763865    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d56c75bc-f03a-4444-a433-a9dfef143814" containerName="maas-api"
Sep 11 16:21:07.763914 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.763876    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d56c75bc-f03a-4444-a433-a9dfef143814" containerName="maas-api"
Sep 11 16:21:07.763914 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.763886    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70fa3e6-9860-4a19-9e87-24af505c8e94" containerName="maas-api"
Sep 11 16:21:07.763914 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.763891    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70fa3e6-9860-4a19-9e87-24af505c8e94" containerName="maas-api"
Sep 11 16:21:07.763914 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.763897    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9e8da3f-ec4e-479a-9c52-4725a79b5d2f" containerName="payload-pre-processing"
Sep 11 16:21:07.763914 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.763903    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9e8da3f-ec4e-479a-9c52-4725a79b5d2f" containerName="payload-pre-processing"
Sep 11 16:21:07.763914 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.763916    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b38ee3f-1329-4007-a7d0-92a17af2f503" containerName="main"
Sep 11 16:21:07.764181 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.763921    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b38ee3f-1329-4007-a7d0-92a17af2f503" containerName="main"
Sep 11 16:21:07.764181 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.763933    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6267331-2bd6-479b-957a-656400a77b57" containerName="payload-pre-processing"
Sep 11 16:21:07.764181 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.763938    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6267331-2bd6-479b-957a-656400a77b57" containerName="payload-pre-processing"
Sep 11 16:21:07.766572 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.764653    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b38ee3f-1329-4007-a7d0-92a17af2f503" containerName="main"
Sep 11 16:21:07.766572 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.764694    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70fa3e6-9860-4a19-9e87-24af505c8e94" containerName="maas-api"
Sep 11 16:21:07.766572 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.764712    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d56c75bc-f03a-4444-a433-a9dfef143814" containerName="maas-api"
Sep 11 16:21:07.766572 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.764726    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6267331-2bd6-479b-957a-656400a77b57" containerName="payload-pre-processing"
Sep 11 16:21:07.766572 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.764742    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9e8da3f-ec4e-479a-9c52-4725a79b5d2f" containerName="payload-pre-processing"
Sep 11 16:21:07.771820 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.771787    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9"
Sep 11 16:21:07.778765 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.778737    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-1ec6e6\""
Sep 11 16:21:07.782551 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.782506    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9"]
Sep 11 16:21:07.834217 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.834180    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6"]
Sep 11 16:21:07.837655 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.837633    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6"
Sep 11 16:21:07.838802 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.838780    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4311d371-c6e8-4d66-8bce-fbea05486610-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9\" (UID: \"4311d371-c6e8-4d66-8bce-fbea05486610\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9"
Sep 11 16:21:07.838916 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.838854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bngt\" (UniqueName: \"kubernetes.io/projected/4311d371-c6e8-4d66-8bce-fbea05486610-kube-api-access-5bngt\") pod \"payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9\" (UID: \"4311d371-c6e8-4d66-8bce-fbea05486610\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9"
Sep 11 16:21:07.845278 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.845251    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-1ec6e6-dockercfg-hwfgx\""
Sep 11 16:21:07.857845 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.857813    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6"]
Sep 11 16:21:07.940086 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.940013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bngt\" (UniqueName: \"kubernetes.io/projected/4311d371-c6e8-4d66-8bce-fbea05486610-kube-api-access-5bngt\") pod \"payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9\" (UID: \"4311d371-c6e8-4d66-8bce-fbea05486610\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9"
Sep 11 16:21:07.940285 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.940100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4311d371-c6e8-4d66-8bce-fbea05486610-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9\" (UID: \"4311d371-c6e8-4d66-8bce-fbea05486610\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9"
Sep 11 16:21:07.940285 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.940144    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djn4n\" (UniqueName: \"kubernetes.io/projected/63051b3a-b943-4742-9ee1-a48d2ae7f28a-kube-api-access-djn4n\") pod \"payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6\" (UID: \"63051b3a-b943-4742-9ee1-a48d2ae7f28a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6"
Sep 11 16:21:07.940285 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.940197    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63051b3a-b943-4742-9ee1-a48d2ae7f28a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6\" (UID: \"63051b3a-b943-4742-9ee1-a48d2ae7f28a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6"
Sep 11 16:21:07.940819 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.940798    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4311d371-c6e8-4d66-8bce-fbea05486610-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9\" (UID: \"4311d371-c6e8-4d66-8bce-fbea05486610\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9"
Sep 11 16:21:07.951412 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:07.951379    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bngt\" (UniqueName: \"kubernetes.io/projected/4311d371-c6e8-4d66-8bce-fbea05486610-kube-api-access-5bngt\") pod \"payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9\" (UID: \"4311d371-c6e8-4d66-8bce-fbea05486610\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9"
Sep 11 16:21:08.041452 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:08.041359    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djn4n\" (UniqueName: \"kubernetes.io/projected/63051b3a-b943-4742-9ee1-a48d2ae7f28a-kube-api-access-djn4n\") pod \"payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6\" (UID: \"63051b3a-b943-4742-9ee1-a48d2ae7f28a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6"
Sep 11 16:21:08.041452 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:08.041414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63051b3a-b943-4742-9ee1-a48d2ae7f28a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6\" (UID: \"63051b3a-b943-4742-9ee1-a48d2ae7f28a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6"
Sep 11 16:21:08.042137 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:08.042118    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63051b3a-b943-4742-9ee1-a48d2ae7f28a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6\" (UID: \"63051b3a-b943-4742-9ee1-a48d2ae7f28a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6"
Sep 11 16:21:08.062437 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:08.062408    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djn4n\" (UniqueName: \"kubernetes.io/projected/63051b3a-b943-4742-9ee1-a48d2ae7f28a-kube-api-access-djn4n\") pod \"payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6\" (UID: \"63051b3a-b943-4742-9ee1-a48d2ae7f28a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6"
Sep 11 16:21:08.084972 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:08.084940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9"
Sep 11 16:21:08.148901 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:08.148806    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6"
Sep 11 16:21:08.374441 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:21:08.374407    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4311d371_c6e8_4d66_8bce_fbea05486610.slice/crio-f1de028a9dcd0ea212b0c6dcb8e96f3cbf8f6fa5c1d5cc07994c038de406fc71 WatchSource:0}: Error finding container f1de028a9dcd0ea212b0c6dcb8e96f3cbf8f6fa5c1d5cc07994c038de406fc71: Status 404 returned error can't find the container with id f1de028a9dcd0ea212b0c6dcb8e96f3cbf8f6fa5c1d5cc07994c038de406fc71
Sep 11 16:21:08.375785 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:08.375764    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9"]
Sep 11 16:21:08.460729 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:08.460703    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6"]
Sep 11 16:21:08.462198 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:21:08.462172    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63051b3a_b943_4742_9ee1_a48d2ae7f28a.slice/crio-3d00f8d1c4cfbc1bacfce39afa88e11bb6d1fdfcb684fd8bface169f21f6e1bc WatchSource:0}: Error finding container 3d00f8d1c4cfbc1bacfce39afa88e11bb6d1fdfcb684fd8bface169f21f6e1bc: Status 404 returned error can't find the container with id 3d00f8d1c4cfbc1bacfce39afa88e11bb6d1fdfcb684fd8bface169f21f6e1bc
Sep 11 16:21:08.975005 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:08.974967    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6" event={"ID":"63051b3a-b943-4742-9ee1-a48d2ae7f28a","Type":"ContainerStarted","Data":"5d8bfa8437d4dcd315a07b5e78edbf270b2262ae2609aec0c6ea3a180de04b02"}
Sep 11 16:21:08.975497 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:08.975171    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6"
Sep 11 16:21:08.975497 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:08.975219    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6" event={"ID":"63051b3a-b943-4742-9ee1-a48d2ae7f28a","Type":"ContainerStarted","Data":"3d00f8d1c4cfbc1bacfce39afa88e11bb6d1fdfcb684fd8bface169f21f6e1bc"}
Sep 11 16:21:08.976544 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:08.976522    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9" event={"ID":"4311d371-c6e8-4d66-8bce-fbea05486610","Type":"ContainerStarted","Data":"f9ef5700e890009f9cb3ae755aad4d94ce069dcbda6944eb1725c1b65f62e9db"}
Sep 11 16:21:08.976666 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:08.976557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9" event={"ID":"4311d371-c6e8-4d66-8bce-fbea05486610","Type":"ContainerStarted","Data":"f1de028a9dcd0ea212b0c6dcb8e96f3cbf8f6fa5c1d5cc07994c038de406fc71"}
Sep 11 16:21:08.976666 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:08.976584    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9"
Sep 11 16:21:09.002426 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:09.002372    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6" podStartSLOduration=2.002355725 podStartE2EDuration="2.002355725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:21:09.001860995 +0000 UTC m=+1128.847504552" watchObservedRunningTime="2026-09-11 16:21:09.002355725 +0000 UTC m=+1128.847999280"
Sep 11 16:21:09.049812 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:09.049745    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9" podStartSLOduration=2.049726579 podStartE2EDuration="2.049726579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:21:09.046918665 +0000 UTC m=+1128.892562225" watchObservedRunningTime="2026-09-11 16:21:09.049726579 +0000 UTC m=+1128.895370132"
Sep 11 16:21:19.984399 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:19.984367    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6"
Sep 11 16:21:19.984795 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:19.984583    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9"
Sep 11 16:21:28.220643 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:28.220607    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"]
Sep 11 16:21:28.225481 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:28.225455    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"
Sep 11 16:21:28.237673 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:28.237651    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-3db535-serving-cert\""
Sep 11 16:21:28.270074 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:28.270020    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"]
Sep 11 16:21:28.321439 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:28.321400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pww5k\" (UniqueName: \"kubernetes.io/projected/e6436e81-28cf-444f-98cf-e1d93b43532f-kube-api-access-pww5k\") pod \"maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv\" (UID: \"e6436e81-28cf-444f-98cf-e1d93b43532f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"
Sep 11 16:21:28.321622 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:28.321461    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6436e81-28cf-444f-98cf-e1d93b43532f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv\" (UID: \"e6436e81-28cf-444f-98cf-e1d93b43532f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"
Sep 11 16:21:28.321622 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:28.321530    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6436e81-28cf-444f-98cf-e1d93b43532f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv\" (UID: \"e6436e81-28cf-444f-98cf-e1d93b43532f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"
Sep 11 16:21:28.321622 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:28.321613    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6436e81-28cf-444f-98cf-e1d93b43532f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv\" (UID: \"e6436e81-28cf-444f-98cf-e1d93b43532f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"
Sep 11 16:21:28.422626 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:28.422590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6436e81-28cf-444f-98cf-e1d93b43532f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv\" (UID: \"e6436e81-28cf-444f-98cf-e1d93b43532f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"
Sep 11 16:21:28.422626 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:28.422629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6436e81-28cf-444f-98cf-e1d93b43532f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv\" (UID: \"e6436e81-28cf-444f-98cf-e1d93b43532f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"
Sep 11 16:21:28.422859 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:28.422675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6436e81-28cf-444f-98cf-e1d93b43532f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv\" (UID: \"e6436e81-28cf-444f-98cf-e1d93b43532f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"
Sep 11 16:21:28.422859 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:28.422708    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pww5k\" (UniqueName: \"kubernetes.io/projected/e6436e81-28cf-444f-98cf-e1d93b43532f-kube-api-access-pww5k\") pod \"maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv\" (UID: \"e6436e81-28cf-444f-98cf-e1d93b43532f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"
Sep 11 16:21:28.423277 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:28.423249    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6436e81-28cf-444f-98cf-e1d93b43532f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv\" (UID: \"e6436e81-28cf-444f-98cf-e1d93b43532f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"
Sep 11 16:21:28.425335 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:28.425311    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6436e81-28cf-444f-98cf-e1d93b43532f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv\" (UID: \"e6436e81-28cf-444f-98cf-e1d93b43532f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"
Sep 11 16:21:28.425519 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:28.425498    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6436e81-28cf-444f-98cf-e1d93b43532f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv\" (UID: \"e6436e81-28cf-444f-98cf-e1d93b43532f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"
Sep 11 16:21:28.460786 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:28.460746    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pww5k\" (UniqueName: \"kubernetes.io/projected/e6436e81-28cf-444f-98cf-e1d93b43532f-kube-api-access-pww5k\") pod \"maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv\" (UID: \"e6436e81-28cf-444f-98cf-e1d93b43532f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"
Sep 11 16:21:28.535390 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:28.535294    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"
Sep 11 16:21:28.830790 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:28.830717    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"]
Sep 11 16:21:28.834111 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:21:28.834082    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6436e81_28cf_444f_98cf_e1d93b43532f.slice/crio-af6e0e483e82f3be773e9403c5fdd79d08ad74a7086631b626fd4fde359ab4a5 WatchSource:0}: Error finding container af6e0e483e82f3be773e9403c5fdd79d08ad74a7086631b626fd4fde359ab4a5: Status 404 returned error can't find the container with id af6e0e483e82f3be773e9403c5fdd79d08ad74a7086631b626fd4fde359ab4a5
Sep 11 16:21:29.078267 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:29.078236    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv" event={"ID":"e6436e81-28cf-444f-98cf-e1d93b43532f","Type":"ContainerStarted","Data":"af6e0e483e82f3be773e9403c5fdd79d08ad74a7086631b626fd4fde359ab4a5"}
Sep 11 16:21:30.085063 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:30.085000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv" event={"ID":"e6436e81-28cf-444f-98cf-e1d93b43532f","Type":"ContainerStarted","Data":"c516e46d92c90736db01869b2c1e41718c20a4b62afe5b0dcb44449c7b1a82aa"}
Sep 11 16:21:30.085458 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:30.085172    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"
Sep 11 16:21:30.115243 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:30.115194    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv" podStartSLOduration=1.709371112 podStartE2EDuration="2.1151774s" totalImagesPullingTime="405.806288ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:21:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:21:30.114412859 +0000 UTC m=+1149.960056415" watchObservedRunningTime="2026-09-11 16:21:30.1151774 +0000 UTC m=+1149.960820955"
Sep 11 16:21:36.096605 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:36.096569    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"
Sep 11 16:21:38.746928 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:38.746893    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4"]
Sep 11 16:21:38.750736 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:38.750718    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4"
Sep 11 16:21:38.755333 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:38.755301    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-233d22-dockercfg-dvcl4\""
Sep 11 16:21:38.755442 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:38.755336    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-233d22\""
Sep 11 16:21:38.768596 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:38.768568    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4"]
Sep 11 16:21:38.919046 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:38.918990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p4pv\" (UniqueName: \"kubernetes.io/projected/6897d110-2a89-4930-92f3-c74204aff896-kube-api-access-5p4pv\") pod \"payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4\" (UID: \"6897d110-2a89-4930-92f3-c74204aff896\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4"
Sep 11 16:21:38.919225 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:38.919061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6897d110-2a89-4930-92f3-c74204aff896-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4\" (UID: \"6897d110-2a89-4930-92f3-c74204aff896\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4"
Sep 11 16:21:39.020298 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:39.020213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6897d110-2a89-4930-92f3-c74204aff896-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4\" (UID: \"6897d110-2a89-4930-92f3-c74204aff896\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4"
Sep 11 16:21:39.020298 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:39.020290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p4pv\" (UniqueName: \"kubernetes.io/projected/6897d110-2a89-4930-92f3-c74204aff896-kube-api-access-5p4pv\") pod \"payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4\" (UID: \"6897d110-2a89-4930-92f3-c74204aff896\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4"
Sep 11 16:21:39.020916 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:39.020896    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6897d110-2a89-4930-92f3-c74204aff896-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4\" (UID: \"6897d110-2a89-4930-92f3-c74204aff896\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4"
Sep 11 16:21:39.032240 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:39.032217    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p4pv\" (UniqueName: \"kubernetes.io/projected/6897d110-2a89-4930-92f3-c74204aff896-kube-api-access-5p4pv\") pod \"payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4\" (UID: \"6897d110-2a89-4930-92f3-c74204aff896\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4"
Sep 11 16:21:39.062204 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:39.062156    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4"
Sep 11 16:21:39.342857 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:39.342722    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4"]
Sep 11 16:21:39.345890 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:21:39.345865    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6897d110_2a89_4930_92f3_c74204aff896.slice/crio-6fb1442a77abf1ae6962d8349d4e85b342980387688200cca034026bc8ae1483 WatchSource:0}: Error finding container 6fb1442a77abf1ae6962d8349d4e85b342980387688200cca034026bc8ae1483: Status 404 returned error can't find the container with id 6fb1442a77abf1ae6962d8349d4e85b342980387688200cca034026bc8ae1483
Sep 11 16:21:40.141126 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:40.141083    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4" event={"ID":"6897d110-2a89-4930-92f3-c74204aff896","Type":"ContainerStarted","Data":"40bd62ad586554ce9e48854989897ae7aa14599872e0ce6ea1895ec9b9bb994f"}
Sep 11 16:21:40.141602 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:40.141179    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4" event={"ID":"6897d110-2a89-4930-92f3-c74204aff896","Type":"ContainerStarted","Data":"6fb1442a77abf1ae6962d8349d4e85b342980387688200cca034026bc8ae1483"}
Sep 11 16:21:40.141602 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:40.141235    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4"
Sep 11 16:21:40.244537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:40.244470    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4" podStartSLOduration=2.24444994 podStartE2EDuration="2.24444994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:21:40.243980219 +0000 UTC m=+1160.089623773" watchObservedRunningTime="2026-09-11 16:21:40.24444994 +0000 UTC m=+1160.090093495"
Sep 11 16:21:48.669357 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.669276    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97"]
Sep 11 16:21:48.673028 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.673009    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97"
Sep 11 16:21:48.675673 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.675648    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-7ce5d6\""
Sep 11 16:21:48.686886 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.686861    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97"]
Sep 11 16:21:48.697182 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.697154    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v"]
Sep 11 16:21:48.701418 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.701395    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v"
Sep 11 16:21:48.702688 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.702663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea518d6-18e1-44da-88ca-3e98e28d9c58-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97\" (UID: \"cea518d6-18e1-44da-88ca-3e98e28d9c58\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97"
Sep 11 16:21:48.702802 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.702782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr9xw\" (UniqueName: \"kubernetes.io/projected/cea518d6-18e1-44da-88ca-3e98e28d9c58-kube-api-access-hr9xw\") pod \"payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97\" (UID: \"cea518d6-18e1-44da-88ca-3e98e28d9c58\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97"
Sep 11 16:21:48.705268 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.705252    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-7ce5d6-dockercfg-767xc\""
Sep 11 16:21:48.723727 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.723697    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v"]
Sep 11 16:21:48.804112 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.804083    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e311d34f-1690-4947-bf97-9d291baa69fa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v\" (UID: \"e311d34f-1690-4947-bf97-9d291baa69fa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v"
Sep 11 16:21:48.804284 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.804122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wvhb\" (UniqueName: \"kubernetes.io/projected/e311d34f-1690-4947-bf97-9d291baa69fa-kube-api-access-5wvhb\") pod \"payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v\" (UID: \"e311d34f-1690-4947-bf97-9d291baa69fa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v"
Sep 11 16:21:48.804284 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.804235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr9xw\" (UniqueName: \"kubernetes.io/projected/cea518d6-18e1-44da-88ca-3e98e28d9c58-kube-api-access-hr9xw\") pod \"payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97\" (UID: \"cea518d6-18e1-44da-88ca-3e98e28d9c58\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97"
Sep 11 16:21:48.804284 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.804276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea518d6-18e1-44da-88ca-3e98e28d9c58-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97\" (UID: \"cea518d6-18e1-44da-88ca-3e98e28d9c58\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97"
Sep 11 16:21:48.804904 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.804885    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea518d6-18e1-44da-88ca-3e98e28d9c58-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97\" (UID: \"cea518d6-18e1-44da-88ca-3e98e28d9c58\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97"
Sep 11 16:21:48.814388 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.814351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr9xw\" (UniqueName: \"kubernetes.io/projected/cea518d6-18e1-44da-88ca-3e98e28d9c58-kube-api-access-hr9xw\") pod \"payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97\" (UID: \"cea518d6-18e1-44da-88ca-3e98e28d9c58\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97"
Sep 11 16:21:48.904731 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.904697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e311d34f-1690-4947-bf97-9d291baa69fa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v\" (UID: \"e311d34f-1690-4947-bf97-9d291baa69fa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v"
Sep 11 16:21:48.904731 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.904733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wvhb\" (UniqueName: \"kubernetes.io/projected/e311d34f-1690-4947-bf97-9d291baa69fa-kube-api-access-5wvhb\") pod \"payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v\" (UID: \"e311d34f-1690-4947-bf97-9d291baa69fa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v"
Sep 11 16:21:48.905422 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.905402    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e311d34f-1690-4947-bf97-9d291baa69fa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v\" (UID: \"e311d34f-1690-4947-bf97-9d291baa69fa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v"
Sep 11 16:21:48.914812 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.914778    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wvhb\" (UniqueName: \"kubernetes.io/projected/e311d34f-1690-4947-bf97-9d291baa69fa-kube-api-access-5wvhb\") pod \"payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v\" (UID: \"e311d34f-1690-4947-bf97-9d291baa69fa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v"
Sep 11 16:21:48.983682 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:48.983599    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97"
Sep 11 16:21:49.019993 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:49.019951    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v"
Sep 11 16:21:49.263877 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:49.263852    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97"]
Sep 11 16:21:49.265315 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:21:49.265289    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea518d6_18e1_44da_88ca_3e98e28d9c58.slice/crio-d83a81eb65c13c95fa9a2968bda75f9281c27583e2b0f1cbcb5a85383501a59c WatchSource:0}: Error finding container d83a81eb65c13c95fa9a2968bda75f9281c27583e2b0f1cbcb5a85383501a59c: Status 404 returned error can't find the container with id d83a81eb65c13c95fa9a2968bda75f9281c27583e2b0f1cbcb5a85383501a59c
Sep 11 16:21:49.303054 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:49.303013    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v"]
Sep 11 16:21:49.316225 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:21:49.316191    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode311d34f_1690_4947_bf97_9d291baa69fa.slice/crio-3f81151a867107b4c4dd0a5adec3a6c039bb3598ffd0aa9a86bfec979f41d2c9 WatchSource:0}: Error finding container 3f81151a867107b4c4dd0a5adec3a6c039bb3598ffd0aa9a86bfec979f41d2c9: Status 404 returned error can't find the container with id 3f81151a867107b4c4dd0a5adec3a6c039bb3598ffd0aa9a86bfec979f41d2c9
Sep 11 16:21:50.189541 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:50.189505    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v" event={"ID":"e311d34f-1690-4947-bf97-9d291baa69fa","Type":"ContainerStarted","Data":"825e0b7e6e3539be65d4798c5d7fb0ed350dc1a5cd48c2c41ddf1b614a03a505"}
Sep 11 16:21:50.190017 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:50.189570    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v" event={"ID":"e311d34f-1690-4947-bf97-9d291baa69fa","Type":"ContainerStarted","Data":"3f81151a867107b4c4dd0a5adec3a6c039bb3598ffd0aa9a86bfec979f41d2c9"}
Sep 11 16:21:50.190017 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:50.189602    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v"
Sep 11 16:21:50.191308 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:50.191281    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97" event={"ID":"cea518d6-18e1-44da-88ca-3e98e28d9c58","Type":"ContainerStarted","Data":"91d061d551b95a5039784d3c66bc460aec7d2a99ab18efc321f7d2e84d4007ed"}
Sep 11 16:21:50.191415 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:50.191334    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97" event={"ID":"cea518d6-18e1-44da-88ca-3e98e28d9c58","Type":"ContainerStarted","Data":"d83a81eb65c13c95fa9a2968bda75f9281c27583e2b0f1cbcb5a85383501a59c"}
Sep 11 16:21:50.191415 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:50.191361    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97"
Sep 11 16:21:50.210717 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:50.210663    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v" podStartSLOduration=2.210643847 podStartE2EDuration="2.210643847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:21:50.209999739 +0000 UTC m=+1170.055643292" watchObservedRunningTime="2026-09-11 16:21:50.210643847 +0000 UTC m=+1170.056287405"
Sep 11 16:21:50.232539 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:50.232483    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97" podStartSLOduration=2.232461665 podStartE2EDuration="2.232461665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:21:50.22900262 +0000 UTC m=+1170.074646176" watchObservedRunningTime="2026-09-11 16:21:50.232461665 +0000 UTC m=+1170.078105221"
Sep 11 16:21:51.148171 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:21:51.148134    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4"
Sep 11 16:22:01.201766 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:01.201735    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v"
Sep 11 16:22:01.202389 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:01.202341    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97"
Sep 11 16:22:20.596540 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:20.596514    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/1.log"
Sep 11 16:22:20.599225 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:20.599206    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/1.log"
Sep 11 16:22:20.603321 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:20.603302    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ng2ww_07d57c04-0697-419b-9d3e-ca11761da85c/ovn-acl-logging/0.log"
Sep 11 16:22:20.605626 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:20.605608    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ng2ww_07d57c04-0697-419b-9d3e-ca11761da85c/ovn-acl-logging/0.log"
Sep 11 16:22:25.061130 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:25.061099    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf"]
Sep 11 16:22:25.064951 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:25.064935    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf"
Sep 11 16:22:25.068267 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:25.068246    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0a17a0fd\""
Sep 11 16:22:25.068368 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:25.068349    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-0a17a0fd-dockercfg-v2mhv\""
Sep 11 16:22:25.082306 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:25.082279    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf"]
Sep 11 16:22:25.241311 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:25.241275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwhlr\" (UniqueName: \"kubernetes.io/projected/2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81-kube-api-access-jwhlr\") pod \"payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf\" (UID: \"2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81\") " pod="openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf"
Sep 11 16:22:25.241311 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:25.241311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81-plugins-config-volume\") pod \"payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf\" (UID: \"2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81\") " pod="openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf"
Sep 11 16:22:25.342379 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:25.342286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwhlr\" (UniqueName: \"kubernetes.io/projected/2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81-kube-api-access-jwhlr\") pod \"payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf\" (UID: \"2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81\") " pod="openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf"
Sep 11 16:22:25.342379 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:25.342324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81-plugins-config-volume\") pod \"payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf\" (UID: \"2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81\") " pod="openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf"
Sep 11 16:22:25.343007 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:25.342989    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81-plugins-config-volume\") pod \"payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf\" (UID: \"2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81\") " pod="openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf"
Sep 11 16:22:25.351536 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:25.351502    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwhlr\" (UniqueName: \"kubernetes.io/projected/2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81-kube-api-access-jwhlr\") pod \"payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf\" (UID: \"2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81\") " pod="openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf"
Sep 11 16:22:25.375768 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:25.375742    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf"
Sep 11 16:22:25.652865 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:25.652840    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf"]
Sep 11 16:22:25.655081 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:22:25.655020    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4a32e8_1ed0_48f3_93ed_4bd6208a2d81.slice/crio-74f49fe9f3744b8a4cd1ef9cbc334811e9025ed59166f4f71c5c6fad03f6e817 WatchSource:0}: Error finding container 74f49fe9f3744b8a4cd1ef9cbc334811e9025ed59166f4f71c5c6fad03f6e817: Status 404 returned error can't find the container with id 74f49fe9f3744b8a4cd1ef9cbc334811e9025ed59166f4f71c5c6fad03f6e817
Sep 11 16:22:26.379579 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:26.379543    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf" event={"ID":"2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81","Type":"ContainerStarted","Data":"4e130e02e093ad70cfa715036c783e4f206aacfdbee4285220d160338f3815bc"}
Sep 11 16:22:26.380155 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:26.379596    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf" event={"ID":"2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81","Type":"ContainerStarted","Data":"74f49fe9f3744b8a4cd1ef9cbc334811e9025ed59166f4f71c5c6fad03f6e817"}
Sep 11 16:22:26.380155 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:26.379732    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf"
Sep 11 16:22:26.401655 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:26.401600    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf" podStartSLOduration=1.401579653 podStartE2EDuration="1.401579653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:22:26.39876306 +0000 UTC m=+1206.244406637" watchObservedRunningTime="2026-09-11 16:22:26.401579653 +0000 UTC m=+1206.247223215"
Sep 11 16:22:37.387161 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:22:37.387132    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf"
Sep 11 16:23:03.387384 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.387344    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"]
Sep 11 16:23:03.391265 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.391244    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:03.394818 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.394793    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0a17a0fd\"/\"default-dockercfg-lq7rh\""
Sep 11 16:23:03.394818 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.394804    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0a17a0fd\"/\"e2e-lifecycle-model-0a17a0fd-kserve-self-signed-certs\""
Sep 11 16:23:03.394990 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.394821    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0a17a0fd\"/\"kube-root-ca.crt\""
Sep 11 16:23:03.394990 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.394838    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0a17a0fd\"/\"openshift-service-ca.crt\""
Sep 11 16:23:03.401472 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.401448    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"]
Sep 11 16:23:03.489895 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.489860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-home\") pod \"e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") " pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:03.489895 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.489900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/facabb23-b7f5-4e89-9d18-fd18a7e6332f-tls-certs\") pod \"e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") " pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:03.490206 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.489945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-dshm\") pod \"e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") " pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:03.490206 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.490089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-tmp-dir\") pod \"e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") " pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:03.490206 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.490133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-model-cache\") pod \"e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") " pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:03.490206 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.490167    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b6lt\" (UniqueName: \"kubernetes.io/projected/facabb23-b7f5-4e89-9d18-fd18a7e6332f-kube-api-access-6b6lt\") pod \"e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") " pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:03.591469 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.591438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-home\") pod \"e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") " pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:03.591682 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.591482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/facabb23-b7f5-4e89-9d18-fd18a7e6332f-tls-certs\") pod \"e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") " pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:03.591682 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.591533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-dshm\") pod \"e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") " pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:03.591682 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.591579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-tmp-dir\") pod \"e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") " pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:03.591682 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.591605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-model-cache\") pod \"e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") " pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:03.591682 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.591631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b6lt\" (UniqueName: \"kubernetes.io/projected/facabb23-b7f5-4e89-9d18-fd18a7e6332f-kube-api-access-6b6lt\") pod \"e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") " pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:03.591975 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.591935    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-tmp-dir\") pod \"e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") " pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:03.592056 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.591918    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-home\") pod \"e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") " pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:03.592164 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.592142    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-model-cache\") pod \"e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") " pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:03.594082 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.594028    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-dshm\") pod \"e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") " pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:03.594597 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.594575    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/facabb23-b7f5-4e89-9d18-fd18a7e6332f-tls-certs\") pod \"e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") " pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:03.601380 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.601349    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b6lt\" (UniqueName: \"kubernetes.io/projected/facabb23-b7f5-4e89-9d18-fd18a7e6332f-kube-api-access-6b6lt\") pod \"e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") " pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:03.701573 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.701475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:03.979618 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:03.979591    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"]
Sep 11 16:23:03.985527 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:23:03.982299    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfacabb23_b7f5_4e89_9d18_fd18a7e6332f.slice/crio-f333c621cd727882b59c3db34cd4a631e79f89a2de0ff6b385c7dd5030c2fdf1 WatchSource:0}: Error finding container f333c621cd727882b59c3db34cd4a631e79f89a2de0ff6b385c7dd5030c2fdf1: Status 404 returned error can't find the container with id f333c621cd727882b59c3db34cd4a631e79f89a2de0ff6b385c7dd5030c2fdf1
Sep 11 16:23:04.588411 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:04.588370    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c" event={"ID":"facabb23-b7f5-4e89-9d18-fd18a7e6332f","Type":"ContainerStarted","Data":"d030c4e1eac9e50cf0eedc3b663b5afe9fc8c8185f590f02430c860d86d2585f"}
Sep 11 16:23:04.588833 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:04.588426    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c" event={"ID":"facabb23-b7f5-4e89-9d18-fd18a7e6332f","Type":"ContainerStarted","Data":"f333c621cd727882b59c3db34cd4a631e79f89a2de0ff6b385c7dd5030c2fdf1"}
Sep 11 16:23:04.606888 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:04.606845    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c" podStartSLOduration=1.606830196 podStartE2EDuration="1.606830196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:23:04.605265456 +0000 UTC m=+1244.450909012" watchObservedRunningTime="2026-09-11 16:23:04.606830196 +0000 UTC m=+1244.452473750"
Sep 11 16:23:13.701740 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:13.701646    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:13.701740 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:13.701727    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:13.715160 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:13.715134    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:23:14.654707 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:23:14.654669    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:24:01.634488 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:01.634451    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf"]
Sep 11 16:24:01.635125 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:01.634675    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf" podUID="2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81" containerName="payload-processing" containerID="cri-o://4e130e02e093ad70cfa715036c783e4f206aacfdbee4285220d160338f3815bc" gracePeriod=30
Sep 11 16:24:01.887406 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:01.887346    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf"
Sep 11 16:24:01.898706 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:01.898679    2577 generic.go:356] "Generic (PLEG): container finished" podID="2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81" containerID="4e130e02e093ad70cfa715036c783e4f206aacfdbee4285220d160338f3815bc" exitCode=0
Sep 11 16:24:01.898860 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:01.898732    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf"
Sep 11 16:24:01.898860 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:01.898756    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf" event={"ID":"2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81","Type":"ContainerDied","Data":"4e130e02e093ad70cfa715036c783e4f206aacfdbee4285220d160338f3815bc"}
Sep 11 16:24:01.898860 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:01.898816    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf" event={"ID":"2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81","Type":"ContainerDied","Data":"74f49fe9f3744b8a4cd1ef9cbc334811e9025ed59166f4f71c5c6fad03f6e817"}
Sep 11 16:24:01.898860 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:01.898838    2577 scope.go:117] "RemoveContainer" containerID="4e130e02e093ad70cfa715036c783e4f206aacfdbee4285220d160338f3815bc"
Sep 11 16:24:01.908220 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:01.908119    2577 scope.go:117] "RemoveContainer" containerID="4e130e02e093ad70cfa715036c783e4f206aacfdbee4285220d160338f3815bc"
Sep 11 16:24:01.908517 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:24:01.908477    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e130e02e093ad70cfa715036c783e4f206aacfdbee4285220d160338f3815bc\": container with ID starting with 4e130e02e093ad70cfa715036c783e4f206aacfdbee4285220d160338f3815bc not found: ID does not exist" containerID="4e130e02e093ad70cfa715036c783e4f206aacfdbee4285220d160338f3815bc"
Sep 11 16:24:01.908629 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:01.908512    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e130e02e093ad70cfa715036c783e4f206aacfdbee4285220d160338f3815bc"} err="failed to get container status \"4e130e02e093ad70cfa715036c783e4f206aacfdbee4285220d160338f3815bc\": rpc error: code = NotFound desc = could not find container \"4e130e02e093ad70cfa715036c783e4f206aacfdbee4285220d160338f3815bc\": container with ID starting with 4e130e02e093ad70cfa715036c783e4f206aacfdbee4285220d160338f3815bc not found: ID does not exist"
Sep 11 16:24:02.015607 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:02.015560    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81-plugins-config-volume\") pod \"2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81\" (UID: \"2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81\") "
Sep 11 16:24:02.015802 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:02.015671    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwhlr\" (UniqueName: \"kubernetes.io/projected/2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81-kube-api-access-jwhlr\") pod \"2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81\" (UID: \"2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81\") "
Sep 11 16:24:02.016063 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:02.016005    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81" (UID: "2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:24:02.017950 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:02.017924    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81-kube-api-access-jwhlr" (OuterVolumeSpecName: "kube-api-access-jwhlr") pod "2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81" (UID: "2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81"). InnerVolumeSpecName "kube-api-access-jwhlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:24:02.116947 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:02.116898    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:24:02.116947 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:02.116937    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwhlr\" (UniqueName: \"kubernetes.io/projected/2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81-kube-api-access-jwhlr\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:24:02.222712 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:02.222682    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf"]
Sep 11 16:24:02.230301 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:02.230276    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0a17a0fd-5f8d88c85-dc5vf"]
Sep 11 16:24:02.633516 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:02.633484    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81" path="/var/lib/kubelet/pods/2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81/volumes"
Sep 11 16:24:09.941947 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:09.941904    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"]
Sep 11 16:24:24.963537 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:24.963487    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c" podUID="facabb23-b7f5-4e89-9d18-fd18a7e6332f" containerName="main" containerID="cri-o://d030c4e1eac9e50cf0eedc3b663b5afe9fc8c8185f590f02430c860d86d2585f" gracePeriod=45
Sep 11 16:24:25.230165 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:25.230101    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:24:25.327728 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:25.327688    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/facabb23-b7f5-4e89-9d18-fd18a7e6332f-tls-certs\") pod \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") "
Sep 11 16:24:25.327893 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:25.327749    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-model-cache\") pod \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") "
Sep 11 16:24:25.327893 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:25.327806    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-dshm\") pod \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") "
Sep 11 16:24:25.327893 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:25.327875    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-tmp-dir\") pod \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") "
Sep 11 16:24:25.328132 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:25.328102    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-model-cache" (OuterVolumeSpecName: "model-cache") pod "facabb23-b7f5-4e89-9d18-fd18a7e6332f" (UID: "facabb23-b7f5-4e89-9d18-fd18a7e6332f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:24:25.328216 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:25.328145    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "facabb23-b7f5-4e89-9d18-fd18a7e6332f" (UID: "facabb23-b7f5-4e89-9d18-fd18a7e6332f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:24:25.328216 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:25.328201    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-home\") pod \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") "
Sep 11 16:24:25.328317 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:25.328242    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6b6lt\" (UniqueName: \"kubernetes.io/projected/facabb23-b7f5-4e89-9d18-fd18a7e6332f-kube-api-access-6b6lt\") pod \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\" (UID: \"facabb23-b7f5-4e89-9d18-fd18a7e6332f\") "
Sep 11 16:24:25.328317 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:25.328246    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-home" (OuterVolumeSpecName: "home") pod "facabb23-b7f5-4e89-9d18-fd18a7e6332f" (UID: "facabb23-b7f5-4e89-9d18-fd18a7e6332f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:24:25.328645 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:25.328574    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-tmp-dir\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:24:25.328645 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:25.328596    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-home\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:24:25.328645 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:25.328609    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-model-cache\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:24:25.330380 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:25.330320    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-dshm" (OuterVolumeSpecName: "dshm") pod "facabb23-b7f5-4e89-9d18-fd18a7e6332f" (UID: "facabb23-b7f5-4e89-9d18-fd18a7e6332f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:24:25.330469 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:25.330436    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/facabb23-b7f5-4e89-9d18-fd18a7e6332f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "facabb23-b7f5-4e89-9d18-fd18a7e6332f" (UID: "facabb23-b7f5-4e89-9d18-fd18a7e6332f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:24:25.330629 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:25.330607    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/facabb23-b7f5-4e89-9d18-fd18a7e6332f-kube-api-access-6b6lt" (OuterVolumeSpecName: "kube-api-access-6b6lt") pod "facabb23-b7f5-4e89-9d18-fd18a7e6332f" (UID: "facabb23-b7f5-4e89-9d18-fd18a7e6332f"). InnerVolumeSpecName "kube-api-access-6b6lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:24:25.429441 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:25.429409    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b6lt\" (UniqueName: \"kubernetes.io/projected/facabb23-b7f5-4e89-9d18-fd18a7e6332f-kube-api-access-6b6lt\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:24:25.429441 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:25.429436    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/facabb23-b7f5-4e89-9d18-fd18a7e6332f-tls-certs\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:24:25.429441 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:25.429446    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/facabb23-b7f5-4e89-9d18-fd18a7e6332f-dshm\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:24:26.028555 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:26.028518    2577 generic.go:356] "Generic (PLEG): container finished" podID="facabb23-b7f5-4e89-9d18-fd18a7e6332f" containerID="d030c4e1eac9e50cf0eedc3b663b5afe9fc8c8185f590f02430c860d86d2585f" exitCode=0
Sep 11 16:24:26.028943 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:26.028567    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"
Sep 11 16:24:26.028943 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:26.028596    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c" event={"ID":"facabb23-b7f5-4e89-9d18-fd18a7e6332f","Type":"ContainerDied","Data":"d030c4e1eac9e50cf0eedc3b663b5afe9fc8c8185f590f02430c860d86d2585f"}
Sep 11 16:24:26.028943 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:26.028638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c" event={"ID":"facabb23-b7f5-4e89-9d18-fd18a7e6332f","Type":"ContainerDied","Data":"f333c621cd727882b59c3db34cd4a631e79f89a2de0ff6b385c7dd5030c2fdf1"}
Sep 11 16:24:26.028943 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:26.028656    2577 scope.go:117] "RemoveContainer" containerID="d030c4e1eac9e50cf0eedc3b663b5afe9fc8c8185f590f02430c860d86d2585f"
Sep 11 16:24:26.038629 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:26.038610    2577 scope.go:117] "RemoveContainer" containerID="d030c4e1eac9e50cf0eedc3b663b5afe9fc8c8185f590f02430c860d86d2585f"
Sep 11 16:24:26.038907 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:24:26.038890    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d030c4e1eac9e50cf0eedc3b663b5afe9fc8c8185f590f02430c860d86d2585f\": container with ID starting with d030c4e1eac9e50cf0eedc3b663b5afe9fc8c8185f590f02430c860d86d2585f not found: ID does not exist" containerID="d030c4e1eac9e50cf0eedc3b663b5afe9fc8c8185f590f02430c860d86d2585f"
Sep 11 16:24:26.038964 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:26.038916    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d030c4e1eac9e50cf0eedc3b663b5afe9fc8c8185f590f02430c860d86d2585f"} err="failed to get container status \"d030c4e1eac9e50cf0eedc3b663b5afe9fc8c8185f590f02430c860d86d2585f\": rpc error: code = NotFound desc = could not find container \"d030c4e1eac9e50cf0eedc3b663b5afe9fc8c8185f590f02430c860d86d2585f\": container with ID starting with d030c4e1eac9e50cf0eedc3b663b5afe9fc8c8185f590f02430c860d86d2585f not found: ID does not exist"
Sep 11 16:24:26.054542 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:26.054511    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"]
Sep 11 16:24:26.058506 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:26.058483    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-0a17a0fd/e2e-lifecycle-model-0a17a0fd-kserve-8475994db7-zdb6c"]
Sep 11 16:24:26.632582 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:24:26.632547    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="facabb23-b7f5-4e89-9d18-fd18a7e6332f" path="/var/lib/kubelet/pods/facabb23-b7f5-4e89-9d18-fd18a7e6332f/volumes"
Sep 11 16:26:02.583827 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.583791    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"]
Sep 11 16:26:02.584469 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.584233    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="facabb23-b7f5-4e89-9d18-fd18a7e6332f" containerName="main"
Sep 11 16:26:02.584469 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.584247    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="facabb23-b7f5-4e89-9d18-fd18a7e6332f" containerName="main"
Sep 11 16:26:02.584469 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.584269    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81" containerName="payload-processing"
Sep 11 16:26:02.584469 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.584275    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81" containerName="payload-processing"
Sep 11 16:26:02.584469 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.584359    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="facabb23-b7f5-4e89-9d18-fd18a7e6332f" containerName="main"
Sep 11 16:26:02.584469 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.584367    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a4a32e8-1ed0-48f3-93ed-4bd6208a2d81" containerName="payload-processing"
Sep 11 16:26:02.587466 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.587447    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"
Sep 11 16:26:02.597602 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.597579    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-60e450-serving-cert\""
Sep 11 16:26:02.603084 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.603058    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-openshift-service-ca\") pod \"maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z\" (UID: \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"
Sep 11 16:26:02.603192 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.603096    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt4js\" (UniqueName: \"kubernetes.io/projected/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-kube-api-access-kt4js\") pod \"maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z\" (UID: \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"
Sep 11 16:26:02.603192 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.603123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-metrics-certs\") pod \"maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z\" (UID: \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"
Sep 11 16:26:02.603192 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.603172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-maas-api-tls\") pod \"maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z\" (UID: \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"
Sep 11 16:26:02.704100 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.704030    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-openshift-service-ca\") pod \"maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z\" (UID: \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"
Sep 11 16:26:02.704100 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.704101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt4js\" (UniqueName: \"kubernetes.io/projected/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-kube-api-access-kt4js\") pod \"maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z\" (UID: \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"
Sep 11 16:26:02.704306 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.704121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-metrics-certs\") pod \"maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z\" (UID: \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"
Sep 11 16:26:02.704306 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.704141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-maas-api-tls\") pod \"maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z\" (UID: \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"
Sep 11 16:26:02.705024 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:26:02.704991    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-serving-cert: secret "maas-api-e2e-api-a-60e450-serving-cert" not found
Sep 11 16:26:02.705178 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:26:02.705104    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-maas-api-tls podName:09cc4c8f-cf9d-47f1-9d40-d9dac7087cab nodeName:}" failed. No retries permitted until 2026-09-11 16:26:03.205082915 +0000 UTC m=+1423.050726456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-maas-api-tls") pod "maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z" (UID: "09cc4c8f-cf9d-47f1-9d40-d9dac7087cab") : secret "maas-api-e2e-api-a-60e450-serving-cert" not found
Sep 11 16:26:02.705252 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.705222    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-openshift-service-ca\") pod \"maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z\" (UID: \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"
Sep 11 16:26:02.707183 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.707156    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"]
Sep 11 16:26:02.707422 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.707403    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-metrics-certs\") pod \"maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z\" (UID: \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"
Sep 11 16:26:02.729975 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:02.729929    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt4js\" (UniqueName: \"kubernetes.io/projected/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-kube-api-access-kt4js\") pod \"maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z\" (UID: \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"
Sep 11 16:26:03.208451 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:03.208413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-maas-api-tls\") pod \"maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z\" (UID: \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"
Sep 11 16:26:03.211221 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:03.211191    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-maas-api-tls\") pod \"maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z\" (UID: \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"
Sep 11 16:26:03.497358 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:03.497269    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"
Sep 11 16:26:03.802386 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:03.802361    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"]
Sep 11 16:26:03.807174 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:03.807151    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:26:03.924024 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:03.923989    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh"]
Sep 11 16:26:03.928881 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:03.928853    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh"
Sep 11 16:26:03.932108 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:03.932087    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-60e450-dockercfg-j5kql\""
Sep 11 16:26:03.932228 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:03.932211    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-60e450\""
Sep 11 16:26:03.954646 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:03.954608    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh"]
Sep 11 16:26:04.015111 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:04.015083    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8d08e0-f27a-4e22-bf6d-538459f55883-plugins-config-volume\") pod \"payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh\" (UID: \"0b8d08e0-f27a-4e22-bf6d-538459f55883\") " pod="openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh"
Sep 11 16:26:04.015272 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:04.015149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9szr\" (UniqueName: \"kubernetes.io/projected/0b8d08e0-f27a-4e22-bf6d-538459f55883-kube-api-access-h9szr\") pod \"payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh\" (UID: \"0b8d08e0-f27a-4e22-bf6d-538459f55883\") " pod="openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh"
Sep 11 16:26:04.116680 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:04.116632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8d08e0-f27a-4e22-bf6d-538459f55883-plugins-config-volume\") pod \"payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh\" (UID: \"0b8d08e0-f27a-4e22-bf6d-538459f55883\") " pod="openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh"
Sep 11 16:26:04.116861 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:04.116718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9szr\" (UniqueName: \"kubernetes.io/projected/0b8d08e0-f27a-4e22-bf6d-538459f55883-kube-api-access-h9szr\") pod \"payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh\" (UID: \"0b8d08e0-f27a-4e22-bf6d-538459f55883\") " pod="openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh"
Sep 11 16:26:04.117408 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:04.117386    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8d08e0-f27a-4e22-bf6d-538459f55883-plugins-config-volume\") pod \"payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh\" (UID: \"0b8d08e0-f27a-4e22-bf6d-538459f55883\") " pod="openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh"
Sep 11 16:26:04.128184 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:04.128146    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9szr\" (UniqueName: \"kubernetes.io/projected/0b8d08e0-f27a-4e22-bf6d-538459f55883-kube-api-access-h9szr\") pod \"payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh\" (UID: \"0b8d08e0-f27a-4e22-bf6d-538459f55883\") " pod="openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh"
Sep 11 16:26:04.240142 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:04.240114    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh"
Sep 11 16:26:04.548640 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:04.548539    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z" event={"ID":"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab","Type":"ContainerStarted","Data":"2457a105504dfd8eb1b5325f5de77ca17c6ced77de8cf57cd4f585571361801e"}
Sep 11 16:26:04.548640 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:04.548607    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z" event={"ID":"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab","Type":"ContainerStarted","Data":"ee631c9e624cbe16ca52217548aa6e5af607e22864eb55b7e7589f0d66ce8184"}
Sep 11 16:26:04.548834 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:04.548731    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"
Sep 11 16:26:04.552896 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:26:04.552839    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8d08e0_f27a_4e22_bf6d_538459f55883.slice/crio-e45f002007b00bfa6cf67a8ed5149c11382a10c586fc65ec453ace62e8c8257e WatchSource:0}: Error finding container e45f002007b00bfa6cf67a8ed5149c11382a10c586fc65ec453ace62e8c8257e: Status 404 returned error can't find the container with id e45f002007b00bfa6cf67a8ed5149c11382a10c586fc65ec453ace62e8c8257e
Sep 11 16:26:04.560158 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:04.556360    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh"]
Sep 11 16:26:05.555964 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:05.555920    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh" event={"ID":"0b8d08e0-f27a-4e22-bf6d-538459f55883","Type":"ContainerStarted","Data":"7116c44369828871a9a7b959f25653b36ca24218a2dd976d65e236d38dac57ff"}
Sep 11 16:26:05.556421 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:05.555990    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh" event={"ID":"0b8d08e0-f27a-4e22-bf6d-538459f55883","Type":"ContainerStarted","Data":"e45f002007b00bfa6cf67a8ed5149c11382a10c586fc65ec453ace62e8c8257e"}
Sep 11 16:26:05.556421 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:05.556146    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh"
Sep 11 16:26:05.583006 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:05.582947    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh" podStartSLOduration=2.582926252 podStartE2EDuration="2.582926252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:26:05.582706069 +0000 UTC m=+1425.428349615" watchObservedRunningTime="2026-09-11 16:26:05.582926252 +0000 UTC m=+1425.428569808"
Sep 11 16:26:05.583970 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:05.583933    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z" podStartSLOduration=3.179116993 podStartE2EDuration="3.58392099s" totalImagesPullingTime="404.803997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:26:04.596929499 +0000 UTC m=+1424.442573055" watchObservedRunningTime="2026-09-11 16:26:05.58392099 +0000 UTC m=+1425.429564547"
Sep 11 16:26:10.561565 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:10.561536    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"
Sep 11 16:26:16.564816 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:16.564787    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh"
Sep 11 16:26:32.502707 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:32.502665    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt"]
Sep 11 16:26:32.505776 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:32.505759    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt"
Sep 11 16:26:32.509448 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:32.509416    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4336c5\""
Sep 11 16:26:32.515770 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:32.515733    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4336c5-dockercfg-4zlmw\""
Sep 11 16:26:32.522692 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:32.522658    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt"]
Sep 11 16:26:32.696699 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:32.696665    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mklxp\" (UniqueName: \"kubernetes.io/projected/69a2d71d-792d-49e6-82a9-14de48f1056c-kube-api-access-mklxp\") pod \"payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt\" (UID: \"69a2d71d-792d-49e6-82a9-14de48f1056c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt"
Sep 11 16:26:32.696906 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:32.696723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a2d71d-792d-49e6-82a9-14de48f1056c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt\" (UID: \"69a2d71d-792d-49e6-82a9-14de48f1056c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt"
Sep 11 16:26:32.797474 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:32.797381    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mklxp\" (UniqueName: \"kubernetes.io/projected/69a2d71d-792d-49e6-82a9-14de48f1056c-kube-api-access-mklxp\") pod \"payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt\" (UID: \"69a2d71d-792d-49e6-82a9-14de48f1056c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt"
Sep 11 16:26:32.797643 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:32.797478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a2d71d-792d-49e6-82a9-14de48f1056c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt\" (UID: \"69a2d71d-792d-49e6-82a9-14de48f1056c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt"
Sep 11 16:26:32.798220 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:32.798201    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a2d71d-792d-49e6-82a9-14de48f1056c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt\" (UID: \"69a2d71d-792d-49e6-82a9-14de48f1056c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt"
Sep 11 16:26:32.808128 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:32.808096    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mklxp\" (UniqueName: \"kubernetes.io/projected/69a2d71d-792d-49e6-82a9-14de48f1056c-kube-api-access-mklxp\") pod \"payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt\" (UID: \"69a2d71d-792d-49e6-82a9-14de48f1056c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt"
Sep 11 16:26:32.816707 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:32.816678    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt"
Sep 11 16:26:33.107050 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:33.107010    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt"]
Sep 11 16:26:33.110139 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:26:33.110105    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a2d71d_792d_49e6_82a9_14de48f1056c.slice/crio-f6630bc4a89cdd8a504356f98d4bafb0b33c735a1a16ec3bb69d399494cbf7ce WatchSource:0}: Error finding container f6630bc4a89cdd8a504356f98d4bafb0b33c735a1a16ec3bb69d399494cbf7ce: Status 404 returned error can't find the container with id f6630bc4a89cdd8a504356f98d4bafb0b33c735a1a16ec3bb69d399494cbf7ce
Sep 11 16:26:33.701388 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:33.701346    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt" event={"ID":"69a2d71d-792d-49e6-82a9-14de48f1056c","Type":"ContainerStarted","Data":"e64b131548ea365d89644487fb79486be2ff4e39ffea2a47978281ced8b99e9f"}
Sep 11 16:26:33.701777 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:33.701421    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt" event={"ID":"69a2d71d-792d-49e6-82a9-14de48f1056c","Type":"ContainerStarted","Data":"f6630bc4a89cdd8a504356f98d4bafb0b33c735a1a16ec3bb69d399494cbf7ce"}
Sep 11 16:26:33.701777 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:33.701508    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt"
Sep 11 16:26:33.722245 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:33.722189    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt" podStartSLOduration=1.722172272 podStartE2EDuration="1.722172272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:26:33.719921305 +0000 UTC m=+1453.565564858" watchObservedRunningTime="2026-09-11 16:26:33.722172272 +0000 UTC m=+1453.567815826"
Sep 11 16:26:44.708928 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:26:44.708892    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt"
Sep 11 16:27:14.157231 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.157189    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"]
Sep 11 16:27:14.157696 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.157437    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z" podUID="09cc4c8f-cf9d-47f1-9d40-d9dac7087cab" containerName="maas-api" containerID="cri-o://2457a105504dfd8eb1b5325f5de77ca17c6ced77de8cf57cd4f585571361801e" gracePeriod=30
Sep 11 16:27:14.218553 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.218522    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh"]
Sep 11 16:27:14.218844 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.218796    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh" podUID="0b8d08e0-f27a-4e22-bf6d-538459f55883" containerName="payload-processing" containerID="cri-o://7116c44369828871a9a7b959f25653b36ca24218a2dd976d65e236d38dac57ff" gracePeriod=30
Sep 11 16:27:14.469841 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.469814    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"
Sep 11 16:27:14.509342 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.509311    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh"
Sep 11 16:27:14.572415 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.572382    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-openshift-service-ca\") pod \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\" (UID: \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\") "
Sep 11 16:27:14.572415 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.572414    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt4js\" (UniqueName: \"kubernetes.io/projected/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-kube-api-access-kt4js\") pod \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\" (UID: \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\") "
Sep 11 16:27:14.572680 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.572432    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9szr\" (UniqueName: \"kubernetes.io/projected/0b8d08e0-f27a-4e22-bf6d-538459f55883-kube-api-access-h9szr\") pod \"0b8d08e0-f27a-4e22-bf6d-538459f55883\" (UID: \"0b8d08e0-f27a-4e22-bf6d-538459f55883\") "
Sep 11 16:27:14.572680 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.572459    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-metrics-certs\") pod \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\" (UID: \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\") "
Sep 11 16:27:14.572680 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.572560    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-maas-api-tls\") pod \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\" (UID: \"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab\") "
Sep 11 16:27:14.572680 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.572622    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8d08e0-f27a-4e22-bf6d-538459f55883-plugins-config-volume\") pod \"0b8d08e0-f27a-4e22-bf6d-538459f55883\" (UID: \"0b8d08e0-f27a-4e22-bf6d-538459f55883\") "
Sep 11 16:27:14.572887 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.572847    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "09cc4c8f-cf9d-47f1-9d40-d9dac7087cab" (UID: "09cc4c8f-cf9d-47f1-9d40-d9dac7087cab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:27:14.572975 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.572942    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-openshift-service-ca\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:27:14.573162 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.573120    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b8d08e0-f27a-4e22-bf6d-538459f55883-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0b8d08e0-f27a-4e22-bf6d-538459f55883" (UID: "0b8d08e0-f27a-4e22-bf6d-538459f55883"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:27:14.574706 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.574684    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "09cc4c8f-cf9d-47f1-9d40-d9dac7087cab" (UID: "09cc4c8f-cf9d-47f1-9d40-d9dac7087cab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:27:14.574804 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.574734    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "09cc4c8f-cf9d-47f1-9d40-d9dac7087cab" (UID: "09cc4c8f-cf9d-47f1-9d40-d9dac7087cab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:27:14.575156 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.575130    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8d08e0-f27a-4e22-bf6d-538459f55883-kube-api-access-h9szr" (OuterVolumeSpecName: "kube-api-access-h9szr") pod "0b8d08e0-f27a-4e22-bf6d-538459f55883" (UID: "0b8d08e0-f27a-4e22-bf6d-538459f55883"). InnerVolumeSpecName "kube-api-access-h9szr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:27:14.575207 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.575166    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-kube-api-access-kt4js" (OuterVolumeSpecName: "kube-api-access-kt4js") pod "09cc4c8f-cf9d-47f1-9d40-d9dac7087cab" (UID: "09cc4c8f-cf9d-47f1-9d40-d9dac7087cab"). InnerVolumeSpecName "kube-api-access-kt4js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:27:14.674267 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.674235    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-maas-api-tls\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:27:14.674267 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.674262    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8d08e0-f27a-4e22-bf6d-538459f55883-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:27:14.674456 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.674274    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt4js\" (UniqueName: \"kubernetes.io/projected/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-kube-api-access-kt4js\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:27:14.674456 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.674292    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9szr\" (UniqueName: \"kubernetes.io/projected/0b8d08e0-f27a-4e22-bf6d-538459f55883-kube-api-access-h9szr\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:27:14.674456 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.674303    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab-metrics-certs\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:27:14.917104 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.917070    2577 generic.go:356] "Generic (PLEG): container finished" podID="09cc4c8f-cf9d-47f1-9d40-d9dac7087cab" containerID="2457a105504dfd8eb1b5325f5de77ca17c6ced77de8cf57cd4f585571361801e" exitCode=0
Sep 11 16:27:14.917291 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.917155    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"
Sep 11 16:27:14.917291 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.917170    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z" event={"ID":"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab","Type":"ContainerDied","Data":"2457a105504dfd8eb1b5325f5de77ca17c6ced77de8cf57cd4f585571361801e"}
Sep 11 16:27:14.917291 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.917221    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z" event={"ID":"09cc4c8f-cf9d-47f1-9d40-d9dac7087cab","Type":"ContainerDied","Data":"ee631c9e624cbe16ca52217548aa6e5af607e22864eb55b7e7589f0d66ce8184"}
Sep 11 16:27:14.917291 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.917250    2577 scope.go:117] "RemoveContainer" containerID="2457a105504dfd8eb1b5325f5de77ca17c6ced77de8cf57cd4f585571361801e"
Sep 11 16:27:14.918748 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.918723    2577 generic.go:356] "Generic (PLEG): container finished" podID="0b8d08e0-f27a-4e22-bf6d-538459f55883" containerID="7116c44369828871a9a7b959f25653b36ca24218a2dd976d65e236d38dac57ff" exitCode=0
Sep 11 16:27:14.918856 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.918763    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh" event={"ID":"0b8d08e0-f27a-4e22-bf6d-538459f55883","Type":"ContainerDied","Data":"7116c44369828871a9a7b959f25653b36ca24218a2dd976d65e236d38dac57ff"}
Sep 11 16:27:14.918856 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.918783    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh"
Sep 11 16:27:14.918856 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.918818    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh" event={"ID":"0b8d08e0-f27a-4e22-bf6d-538459f55883","Type":"ContainerDied","Data":"e45f002007b00bfa6cf67a8ed5149c11382a10c586fc65ec453ace62e8c8257e"}
Sep 11 16:27:14.926880 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.926863    2577 scope.go:117] "RemoveContainer" containerID="2457a105504dfd8eb1b5325f5de77ca17c6ced77de8cf57cd4f585571361801e"
Sep 11 16:27:14.927248 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:27:14.927227    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2457a105504dfd8eb1b5325f5de77ca17c6ced77de8cf57cd4f585571361801e\": container with ID starting with 2457a105504dfd8eb1b5325f5de77ca17c6ced77de8cf57cd4f585571361801e not found: ID does not exist" containerID="2457a105504dfd8eb1b5325f5de77ca17c6ced77de8cf57cd4f585571361801e"
Sep 11 16:27:14.927333 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.927255    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2457a105504dfd8eb1b5325f5de77ca17c6ced77de8cf57cd4f585571361801e"} err="failed to get container status \"2457a105504dfd8eb1b5325f5de77ca17c6ced77de8cf57cd4f585571361801e\": rpc error: code = NotFound desc = could not find container \"2457a105504dfd8eb1b5325f5de77ca17c6ced77de8cf57cd4f585571361801e\": container with ID starting with 2457a105504dfd8eb1b5325f5de77ca17c6ced77de8cf57cd4f585571361801e not found: ID does not exist"
Sep 11 16:27:14.927333 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.927271    2577 scope.go:117] "RemoveContainer" containerID="7116c44369828871a9a7b959f25653b36ca24218a2dd976d65e236d38dac57ff"
Sep 11 16:27:14.935189 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.935167    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"]
Sep 11 16:27:14.935550 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.935538    2577 scope.go:117] "RemoveContainer" containerID="7116c44369828871a9a7b959f25653b36ca24218a2dd976d65e236d38dac57ff"
Sep 11 16:27:14.935817 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:27:14.935798    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7116c44369828871a9a7b959f25653b36ca24218a2dd976d65e236d38dac57ff\": container with ID starting with 7116c44369828871a9a7b959f25653b36ca24218a2dd976d65e236d38dac57ff not found: ID does not exist" containerID="7116c44369828871a9a7b959f25653b36ca24218a2dd976d65e236d38dac57ff"
Sep 11 16:27:14.935875 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.935829    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7116c44369828871a9a7b959f25653b36ca24218a2dd976d65e236d38dac57ff"} err="failed to get container status \"7116c44369828871a9a7b959f25653b36ca24218a2dd976d65e236d38dac57ff\": rpc error: code = NotFound desc = could not find container \"7116c44369828871a9a7b959f25653b36ca24218a2dd976d65e236d38dac57ff\": container with ID starting with 7116c44369828871a9a7b959f25653b36ca24218a2dd976d65e236d38dac57ff not found: ID does not exist"
Sep 11 16:27:14.939815 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.939796    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-60e450-5cbb98cfb8-25p7z"]
Sep 11 16:27:14.955456 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.955435    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh"]
Sep 11 16:27:14.968642 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:14.964755    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-60e450-cb7f966c-4r4wh"]
Sep 11 16:27:16.632378 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:16.632346    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09cc4c8f-cf9d-47f1-9d40-d9dac7087cab" path="/var/lib/kubelet/pods/09cc4c8f-cf9d-47f1-9d40-d9dac7087cab/volumes"
Sep 11 16:27:16.633018 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:16.632694    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b8d08e0-f27a-4e22-bf6d-538459f55883" path="/var/lib/kubelet/pods/0b8d08e0-f27a-4e22-bf6d-538459f55883/volumes"
Sep 11 16:27:20.635677 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:20.635640    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/1.log"
Sep 11 16:27:20.640108 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:20.640084    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/1.log"
Sep 11 16:27:20.645729 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:20.645699    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ng2ww_07d57c04-0697-419b-9d3e-ca11761da85c/ovn-acl-logging/0.log"
Sep 11 16:27:20.647493 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:20.647473    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ng2ww_07d57c04-0697-419b-9d3e-ca11761da85c/ovn-acl-logging/0.log"
Sep 11 16:27:37.214539 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:37.214492    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt"]
Sep 11 16:27:37.215005 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:37.214784    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt" podUID="69a2d71d-792d-49e6-82a9-14de48f1056c" containerName="payload-pre-processing" containerID="cri-o://e64b131548ea365d89644487fb79486be2ff4e39ffea2a47978281ced8b99e9f" gracePeriod=30
Sep 11 16:27:37.482311 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:37.482282    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt"
Sep 11 16:27:37.580307 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:37.580271    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mklxp\" (UniqueName: \"kubernetes.io/projected/69a2d71d-792d-49e6-82a9-14de48f1056c-kube-api-access-mklxp\") pod \"69a2d71d-792d-49e6-82a9-14de48f1056c\" (UID: \"69a2d71d-792d-49e6-82a9-14de48f1056c\") "
Sep 11 16:27:37.580511 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:37.580370    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a2d71d-792d-49e6-82a9-14de48f1056c-plugins-config-volume\") pod \"69a2d71d-792d-49e6-82a9-14de48f1056c\" (UID: \"69a2d71d-792d-49e6-82a9-14de48f1056c\") "
Sep 11 16:27:37.580778 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:37.580749    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a2d71d-792d-49e6-82a9-14de48f1056c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69a2d71d-792d-49e6-82a9-14de48f1056c" (UID: "69a2d71d-792d-49e6-82a9-14de48f1056c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:27:37.582530 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:37.582504    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a2d71d-792d-49e6-82a9-14de48f1056c-kube-api-access-mklxp" (OuterVolumeSpecName: "kube-api-access-mklxp") pod "69a2d71d-792d-49e6-82a9-14de48f1056c" (UID: "69a2d71d-792d-49e6-82a9-14de48f1056c"). InnerVolumeSpecName "kube-api-access-mklxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:27:37.681358 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:37.681323    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a2d71d-792d-49e6-82a9-14de48f1056c-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:27:37.681358 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:37.681354    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mklxp\" (UniqueName: \"kubernetes.io/projected/69a2d71d-792d-49e6-82a9-14de48f1056c-kube-api-access-mklxp\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:27:38.045023 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:38.044982    2577 generic.go:356] "Generic (PLEG): container finished" podID="69a2d71d-792d-49e6-82a9-14de48f1056c" containerID="e64b131548ea365d89644487fb79486be2ff4e39ffea2a47978281ced8b99e9f" exitCode=0
Sep 11 16:27:38.045252 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:38.045055    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt"
Sep 11 16:27:38.045252 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:38.045065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt" event={"ID":"69a2d71d-792d-49e6-82a9-14de48f1056c","Type":"ContainerDied","Data":"e64b131548ea365d89644487fb79486be2ff4e39ffea2a47978281ced8b99e9f"}
Sep 11 16:27:38.045252 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:38.045128    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt" event={"ID":"69a2d71d-792d-49e6-82a9-14de48f1056c","Type":"ContainerDied","Data":"f6630bc4a89cdd8a504356f98d4bafb0b33c735a1a16ec3bb69d399494cbf7ce"}
Sep 11 16:27:38.045252 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:38.045151    2577 scope.go:117] "RemoveContainer" containerID="e64b131548ea365d89644487fb79486be2ff4e39ffea2a47978281ced8b99e9f"
Sep 11 16:27:38.054269 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:38.054249    2577 scope.go:117] "RemoveContainer" containerID="e64b131548ea365d89644487fb79486be2ff4e39ffea2a47978281ced8b99e9f"
Sep 11 16:27:38.054624 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:27:38.054598    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e64b131548ea365d89644487fb79486be2ff4e39ffea2a47978281ced8b99e9f\": container with ID starting with e64b131548ea365d89644487fb79486be2ff4e39ffea2a47978281ced8b99e9f not found: ID does not exist" containerID="e64b131548ea365d89644487fb79486be2ff4e39ffea2a47978281ced8b99e9f"
Sep 11 16:27:38.054711 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:38.054634    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e64b131548ea365d89644487fb79486be2ff4e39ffea2a47978281ced8b99e9f"} err="failed to get container status \"e64b131548ea365d89644487fb79486be2ff4e39ffea2a47978281ced8b99e9f\": rpc error: code = NotFound desc = could not find container \"e64b131548ea365d89644487fb79486be2ff4e39ffea2a47978281ced8b99e9f\": container with ID starting with e64b131548ea365d89644487fb79486be2ff4e39ffea2a47978281ced8b99e9f not found: ID does not exist"
Sep 11 16:27:38.079220 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:38.079190    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt"]
Sep 11 16:27:38.083716 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:38.083684    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4336c5-6dd96cb94-wvgxt"]
Sep 11 16:27:38.633636 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:38.633598    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69a2d71d-792d-49e6-82a9-14de48f1056c" path="/var/lib/kubelet/pods/69a2d71d-792d-49e6-82a9-14de48f1056c/volumes"
Sep 11 16:27:54.869560 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:54.869475    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"]
Sep 11 16:27:54.870149 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:54.869727    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv" podUID="e6436e81-28cf-444f-98cf-e1d93b43532f" containerName="maas-api" containerID="cri-o://c516e46d92c90736db01869b2c1e41718c20a4b62afe5b0dcb44449c7b1a82aa" gracePeriod=30
Sep 11 16:27:55.125261 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.125196    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"
Sep 11 16:27:55.138395 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.138365    2577 generic.go:356] "Generic (PLEG): container finished" podID="e6436e81-28cf-444f-98cf-e1d93b43532f" containerID="c516e46d92c90736db01869b2c1e41718c20a4b62afe5b0dcb44449c7b1a82aa" exitCode=0
Sep 11 16:27:55.138536 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.138410    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"
Sep 11 16:27:55.138536 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.138445    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv" event={"ID":"e6436e81-28cf-444f-98cf-e1d93b43532f","Type":"ContainerDied","Data":"c516e46d92c90736db01869b2c1e41718c20a4b62afe5b0dcb44449c7b1a82aa"}
Sep 11 16:27:55.138536 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.138532    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv" event={"ID":"e6436e81-28cf-444f-98cf-e1d93b43532f","Type":"ContainerDied","Data":"af6e0e483e82f3be773e9403c5fdd79d08ad74a7086631b626fd4fde359ab4a5"}
Sep 11 16:27:55.138665 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.138561    2577 scope.go:117] "RemoveContainer" containerID="c516e46d92c90736db01869b2c1e41718c20a4b62afe5b0dcb44449c7b1a82aa"
Sep 11 16:27:55.148910 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.148884    2577 scope.go:117] "RemoveContainer" containerID="c516e46d92c90736db01869b2c1e41718c20a4b62afe5b0dcb44449c7b1a82aa"
Sep 11 16:27:55.149354 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:27:55.149331    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c516e46d92c90736db01869b2c1e41718c20a4b62afe5b0dcb44449c7b1a82aa\": container with ID starting with c516e46d92c90736db01869b2c1e41718c20a4b62afe5b0dcb44449c7b1a82aa not found: ID does not exist" containerID="c516e46d92c90736db01869b2c1e41718c20a4b62afe5b0dcb44449c7b1a82aa"
Sep 11 16:27:55.149464 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.149365    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c516e46d92c90736db01869b2c1e41718c20a4b62afe5b0dcb44449c7b1a82aa"} err="failed to get container status \"c516e46d92c90736db01869b2c1e41718c20a4b62afe5b0dcb44449c7b1a82aa\": rpc error: code = NotFound desc = could not find container \"c516e46d92c90736db01869b2c1e41718c20a4b62afe5b0dcb44449c7b1a82aa\": container with ID starting with c516e46d92c90736db01869b2c1e41718c20a4b62afe5b0dcb44449c7b1a82aa not found: ID does not exist"
Sep 11 16:27:55.239922 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.239878    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6436e81-28cf-444f-98cf-e1d93b43532f-metrics-certs\") pod \"e6436e81-28cf-444f-98cf-e1d93b43532f\" (UID: \"e6436e81-28cf-444f-98cf-e1d93b43532f\") "
Sep 11 16:27:55.239922 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.239915    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6436e81-28cf-444f-98cf-e1d93b43532f-maas-api-tls\") pod \"e6436e81-28cf-444f-98cf-e1d93b43532f\" (UID: \"e6436e81-28cf-444f-98cf-e1d93b43532f\") "
Sep 11 16:27:55.240181 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.239966    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pww5k\" (UniqueName: \"kubernetes.io/projected/e6436e81-28cf-444f-98cf-e1d93b43532f-kube-api-access-pww5k\") pod \"e6436e81-28cf-444f-98cf-e1d93b43532f\" (UID: \"e6436e81-28cf-444f-98cf-e1d93b43532f\") "
Sep 11 16:27:55.240181 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.240030    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6436e81-28cf-444f-98cf-e1d93b43532f-openshift-service-ca\") pod \"e6436e81-28cf-444f-98cf-e1d93b43532f\" (UID: \"e6436e81-28cf-444f-98cf-e1d93b43532f\") "
Sep 11 16:27:55.240449 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.240413    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6436e81-28cf-444f-98cf-e1d93b43532f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e6436e81-28cf-444f-98cf-e1d93b43532f" (UID: "e6436e81-28cf-444f-98cf-e1d93b43532f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:27:55.242257 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.242232    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6436e81-28cf-444f-98cf-e1d93b43532f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e6436e81-28cf-444f-98cf-e1d93b43532f" (UID: "e6436e81-28cf-444f-98cf-e1d93b43532f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:27:55.242375 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.242283    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6436e81-28cf-444f-98cf-e1d93b43532f-kube-api-access-pww5k" (OuterVolumeSpecName: "kube-api-access-pww5k") pod "e6436e81-28cf-444f-98cf-e1d93b43532f" (UID: "e6436e81-28cf-444f-98cf-e1d93b43532f"). InnerVolumeSpecName "kube-api-access-pww5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:27:55.242375 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.242305    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6436e81-28cf-444f-98cf-e1d93b43532f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e6436e81-28cf-444f-98cf-e1d93b43532f" (UID: "e6436e81-28cf-444f-98cf-e1d93b43532f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:27:55.341580 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.341534    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pww5k\" (UniqueName: \"kubernetes.io/projected/e6436e81-28cf-444f-98cf-e1d93b43532f-kube-api-access-pww5k\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:27:55.341580 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.341573    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6436e81-28cf-444f-98cf-e1d93b43532f-openshift-service-ca\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:27:55.341580 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.341589    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6436e81-28cf-444f-98cf-e1d93b43532f-metrics-certs\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:27:55.341840 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.341601    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6436e81-28cf-444f-98cf-e1d93b43532f-maas-api-tls\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:27:55.463597 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.463566    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"]
Sep 11 16:27:55.469794 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:55.469763    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db535-97dd85565-bs6zv"]
Sep 11 16:27:56.632682 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:27:56.632650    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6436e81-28cf-444f-98cf-e1d93b43532f" path="/var/lib/kubelet/pods/e6436e81-28cf-444f-98cf-e1d93b43532f/volumes"
Sep 11 16:28:13.566713 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:13.566680    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v"]
Sep 11 16:28:13.567236 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:13.566913    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v" podUID="e311d34f-1690-4947-bf97-9d291baa69fa" containerName="payload-processing" containerID="cri-o://825e0b7e6e3539be65d4798c5d7fb0ed350dc1a5cd48c2c41ddf1b614a03a505" gracePeriod=30
Sep 11 16:28:13.582600 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:13.582559    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97"]
Sep 11 16:28:13.582924 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:13.582831    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97" podUID="cea518d6-18e1-44da-88ca-3e98e28d9c58" containerName="payload-pre-processing" containerID="cri-o://91d061d551b95a5039784d3c66bc460aec7d2a99ab18efc321f7d2e84d4007ed" gracePeriod=30
Sep 11 16:28:13.867007 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:13.866979    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v"
Sep 11 16:28:13.870184 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:13.870162    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97"
Sep 11 16:28:14.017138 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.017104    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr9xw\" (UniqueName: \"kubernetes.io/projected/cea518d6-18e1-44da-88ca-3e98e28d9c58-kube-api-access-hr9xw\") pod \"cea518d6-18e1-44da-88ca-3e98e28d9c58\" (UID: \"cea518d6-18e1-44da-88ca-3e98e28d9c58\") "
Sep 11 16:28:14.017138 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.017141    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wvhb\" (UniqueName: \"kubernetes.io/projected/e311d34f-1690-4947-bf97-9d291baa69fa-kube-api-access-5wvhb\") pod \"e311d34f-1690-4947-bf97-9d291baa69fa\" (UID: \"e311d34f-1690-4947-bf97-9d291baa69fa\") "
Sep 11 16:28:14.017391 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.017164    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e311d34f-1690-4947-bf97-9d291baa69fa-plugins-config-volume\") pod \"e311d34f-1690-4947-bf97-9d291baa69fa\" (UID: \"e311d34f-1690-4947-bf97-9d291baa69fa\") "
Sep 11 16:28:14.017391 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.017236    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea518d6-18e1-44da-88ca-3e98e28d9c58-plugins-config-volume\") pod \"cea518d6-18e1-44da-88ca-3e98e28d9c58\" (UID: \"cea518d6-18e1-44da-88ca-3e98e28d9c58\") "
Sep 11 16:28:14.017581 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.017554    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e311d34f-1690-4947-bf97-9d291baa69fa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e311d34f-1690-4947-bf97-9d291baa69fa" (UID: "e311d34f-1690-4947-bf97-9d291baa69fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:28:14.017692 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.017668    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea518d6-18e1-44da-88ca-3e98e28d9c58-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cea518d6-18e1-44da-88ca-3e98e28d9c58" (UID: "cea518d6-18e1-44da-88ca-3e98e28d9c58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:28:14.019472 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.019448    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e311d34f-1690-4947-bf97-9d291baa69fa-kube-api-access-5wvhb" (OuterVolumeSpecName: "kube-api-access-5wvhb") pod "e311d34f-1690-4947-bf97-9d291baa69fa" (UID: "e311d34f-1690-4947-bf97-9d291baa69fa"). InnerVolumeSpecName "kube-api-access-5wvhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:28:14.019572 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.019480    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea518d6-18e1-44da-88ca-3e98e28d9c58-kube-api-access-hr9xw" (OuterVolumeSpecName: "kube-api-access-hr9xw") pod "cea518d6-18e1-44da-88ca-3e98e28d9c58" (UID: "cea518d6-18e1-44da-88ca-3e98e28d9c58"). InnerVolumeSpecName "kube-api-access-hr9xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:28:14.118103 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.118070    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr9xw\" (UniqueName: \"kubernetes.io/projected/cea518d6-18e1-44da-88ca-3e98e28d9c58-kube-api-access-hr9xw\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:28:14.118103 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.118095    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wvhb\" (UniqueName: \"kubernetes.io/projected/e311d34f-1690-4947-bf97-9d291baa69fa-kube-api-access-5wvhb\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:28:14.118103 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.118106    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e311d34f-1690-4947-bf97-9d291baa69fa-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:28:14.118335 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.118115    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea518d6-18e1-44da-88ca-3e98e28d9c58-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:28:14.237069 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.237018    2577 generic.go:356] "Generic (PLEG): container finished" podID="e311d34f-1690-4947-bf97-9d291baa69fa" containerID="825e0b7e6e3539be65d4798c5d7fb0ed350dc1a5cd48c2c41ddf1b614a03a505" exitCode=0
Sep 11 16:28:14.237242 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.237065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v" event={"ID":"e311d34f-1690-4947-bf97-9d291baa69fa","Type":"ContainerDied","Data":"825e0b7e6e3539be65d4798c5d7fb0ed350dc1a5cd48c2c41ddf1b614a03a505"}
Sep 11 16:28:14.237242 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.237089    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v"
Sep 11 16:28:14.237242 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.237118    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v" event={"ID":"e311d34f-1690-4947-bf97-9d291baa69fa","Type":"ContainerDied","Data":"3f81151a867107b4c4dd0a5adec3a6c039bb3598ffd0aa9a86bfec979f41d2c9"}
Sep 11 16:28:14.237242 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.237141    2577 scope.go:117] "RemoveContainer" containerID="825e0b7e6e3539be65d4798c5d7fb0ed350dc1a5cd48c2c41ddf1b614a03a505"
Sep 11 16:28:14.238614 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.238590    2577 generic.go:356] "Generic (PLEG): container finished" podID="cea518d6-18e1-44da-88ca-3e98e28d9c58" containerID="91d061d551b95a5039784d3c66bc460aec7d2a99ab18efc321f7d2e84d4007ed" exitCode=0
Sep 11 16:28:14.238696 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.238643    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97"
Sep 11 16:28:14.238696 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.238680    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97" event={"ID":"cea518d6-18e1-44da-88ca-3e98e28d9c58","Type":"ContainerDied","Data":"91d061d551b95a5039784d3c66bc460aec7d2a99ab18efc321f7d2e84d4007ed"}
Sep 11 16:28:14.238801 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.238735    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97" event={"ID":"cea518d6-18e1-44da-88ca-3e98e28d9c58","Type":"ContainerDied","Data":"d83a81eb65c13c95fa9a2968bda75f9281c27583e2b0f1cbcb5a85383501a59c"}
Sep 11 16:28:14.247875 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.247857    2577 scope.go:117] "RemoveContainer" containerID="825e0b7e6e3539be65d4798c5d7fb0ed350dc1a5cd48c2c41ddf1b614a03a505"
Sep 11 16:28:14.248341 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:28:14.248179    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"825e0b7e6e3539be65d4798c5d7fb0ed350dc1a5cd48c2c41ddf1b614a03a505\": container with ID starting with 825e0b7e6e3539be65d4798c5d7fb0ed350dc1a5cd48c2c41ddf1b614a03a505 not found: ID does not exist" containerID="825e0b7e6e3539be65d4798c5d7fb0ed350dc1a5cd48c2c41ddf1b614a03a505"
Sep 11 16:28:14.248341 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.248208    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"825e0b7e6e3539be65d4798c5d7fb0ed350dc1a5cd48c2c41ddf1b614a03a505"} err="failed to get container status \"825e0b7e6e3539be65d4798c5d7fb0ed350dc1a5cd48c2c41ddf1b614a03a505\": rpc error: code = NotFound desc = could not find container \"825e0b7e6e3539be65d4798c5d7fb0ed350dc1a5cd48c2c41ddf1b614a03a505\": container with ID starting with 825e0b7e6e3539be65d4798c5d7fb0ed350dc1a5cd48c2c41ddf1b614a03a505 not found: ID does not exist"
Sep 11 16:28:14.248341 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.248231    2577 scope.go:117] "RemoveContainer" containerID="91d061d551b95a5039784d3c66bc460aec7d2a99ab18efc321f7d2e84d4007ed"
Sep 11 16:28:14.256439 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.256421    2577 scope.go:117] "RemoveContainer" containerID="91d061d551b95a5039784d3c66bc460aec7d2a99ab18efc321f7d2e84d4007ed"
Sep 11 16:28:14.256725 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:28:14.256708    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d061d551b95a5039784d3c66bc460aec7d2a99ab18efc321f7d2e84d4007ed\": container with ID starting with 91d061d551b95a5039784d3c66bc460aec7d2a99ab18efc321f7d2e84d4007ed not found: ID does not exist" containerID="91d061d551b95a5039784d3c66bc460aec7d2a99ab18efc321f7d2e84d4007ed"
Sep 11 16:28:14.256779 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.256733    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d061d551b95a5039784d3c66bc460aec7d2a99ab18efc321f7d2e84d4007ed"} err="failed to get container status \"91d061d551b95a5039784d3c66bc460aec7d2a99ab18efc321f7d2e84d4007ed\": rpc error: code = NotFound desc = could not find container \"91d061d551b95a5039784d3c66bc460aec7d2a99ab18efc321f7d2e84d4007ed\": container with ID starting with 91d061d551b95a5039784d3c66bc460aec7d2a99ab18efc321f7d2e84d4007ed not found: ID does not exist"
Sep 11 16:28:14.268673 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.268646    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v"]
Sep 11 16:28:14.283170 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.283132    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7ce5d6-6dc88bbb47-27d5v"]
Sep 11 16:28:14.298187 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.298161    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97"]
Sep 11 16:28:14.303061 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.303028    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ce5d6-7f55bfcb5d-59s97"]
Sep 11 16:28:14.632812 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.632778    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cea518d6-18e1-44da-88ca-3e98e28d9c58" path="/var/lib/kubelet/pods/cea518d6-18e1-44da-88ca-3e98e28d9c58/volumes"
Sep 11 16:28:14.633209 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:14.633194    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e311d34f-1690-4947-bf97-9d291baa69fa" path="/var/lib/kubelet/pods/e311d34f-1690-4947-bf97-9d291baa69fa/volumes"
Sep 11 16:28:57.441407 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.441369    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"]
Sep 11 16:28:57.442135 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.441968    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e311d34f-1690-4947-bf97-9d291baa69fa" containerName="payload-processing"
Sep 11 16:28:57.442135 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.441989    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e311d34f-1690-4947-bf97-9d291baa69fa" containerName="payload-processing"
Sep 11 16:28:57.442135 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.442004    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09cc4c8f-cf9d-47f1-9d40-d9dac7087cab" containerName="maas-api"
Sep 11 16:28:57.442135 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.442014    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="09cc4c8f-cf9d-47f1-9d40-d9dac7087cab" containerName="maas-api"
Sep 11 16:28:57.442135 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.442048    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6436e81-28cf-444f-98cf-e1d93b43532f" containerName="maas-api"
Sep 11 16:28:57.442135 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.442071    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6436e81-28cf-444f-98cf-e1d93b43532f" containerName="maas-api"
Sep 11 16:28:57.442135 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.442090    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a2d71d-792d-49e6-82a9-14de48f1056c" containerName="payload-pre-processing"
Sep 11 16:28:57.442135 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.442109    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a2d71d-792d-49e6-82a9-14de48f1056c" containerName="payload-pre-processing"
Sep 11 16:28:57.442135 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.442134    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cea518d6-18e1-44da-88ca-3e98e28d9c58" containerName="payload-pre-processing"
Sep 11 16:28:57.442135 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.442142    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cea518d6-18e1-44da-88ca-3e98e28d9c58" containerName="payload-pre-processing"
Sep 11 16:28:57.442672 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.442155    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b8d08e0-f27a-4e22-bf6d-538459f55883" containerName="payload-processing"
Sep 11 16:28:57.442672 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.442163    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b8d08e0-f27a-4e22-bf6d-538459f55883" containerName="payload-processing"
Sep 11 16:28:57.442934 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.442912    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6436e81-28cf-444f-98cf-e1d93b43532f" containerName="maas-api"
Sep 11 16:28:57.443007 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.442940    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="09cc4c8f-cf9d-47f1-9d40-d9dac7087cab" containerName="maas-api"
Sep 11 16:28:57.443007 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.442952    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a2d71d-792d-49e6-82a9-14de48f1056c" containerName="payload-pre-processing"
Sep 11 16:28:57.443007 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.442963    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cea518d6-18e1-44da-88ca-3e98e28d9c58" containerName="payload-pre-processing"
Sep 11 16:28:57.443007 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.442970    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b8d08e0-f27a-4e22-bf6d-538459f55883" containerName="payload-processing"
Sep 11 16:28:57.443007 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.442984    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e311d34f-1690-4947-bf97-9d291baa69fa" containerName="payload-processing"
Sep 11 16:28:57.445160 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.445138    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"
Sep 11 16:28:57.449614 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.449590    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-9bac29-serving-cert\""
Sep 11 16:28:57.461760 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.461731    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"]
Sep 11 16:28:57.587713 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.587673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-metrics-certs\") pod \"maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz\" (UID: \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"
Sep 11 16:28:57.587923 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.587725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pncpp\" (UniqueName: \"kubernetes.io/projected/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-kube-api-access-pncpp\") pod \"maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz\" (UID: \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"
Sep 11 16:28:57.587923 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.587751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz\" (UID: \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"
Sep 11 16:28:57.587923 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.587876    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz\" (UID: \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"
Sep 11 16:28:57.689265 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.689223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz\" (UID: \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"
Sep 11 16:28:57.689463 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.689298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-metrics-certs\") pod \"maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz\" (UID: \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"
Sep 11 16:28:57.689463 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.689343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pncpp\" (UniqueName: \"kubernetes.io/projected/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-kube-api-access-pncpp\") pod \"maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz\" (UID: \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"
Sep 11 16:28:57.689463 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.689375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz\" (UID: \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"
Sep 11 16:28:57.689463 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:28:57.689386    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-serving-cert: secret "maas-api-e2e-ipp-a-9bac29-serving-cert" not found
Sep 11 16:28:57.689673 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:28:57.689494    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-maas-api-tls podName:d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8 nodeName:}" failed. No retries permitted until 2026-09-11 16:28:58.189470742 +0000 UTC m=+1598.035114291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-maas-api-tls") pod "maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz" (UID: "d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8") : secret "maas-api-e2e-ipp-a-9bac29-serving-cert" not found
Sep 11 16:28:57.690152 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.690125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz\" (UID: \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"
Sep 11 16:28:57.691908 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.691861    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-metrics-certs\") pod \"maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz\" (UID: \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"
Sep 11 16:28:57.702020 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:57.701995    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pncpp\" (UniqueName: \"kubernetes.io/projected/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-kube-api-access-pncpp\") pod \"maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz\" (UID: \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"
Sep 11 16:28:58.193640 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:58.193602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz\" (UID: \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"
Sep 11 16:28:58.196303 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:58.196271    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz\" (UID: \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"
Sep 11 16:28:58.355952 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:58.355909    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"
Sep 11 16:28:58.642426 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:58.642401    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"]
Sep 11 16:28:58.643942 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:28:58.643917    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34c6fe2_c2a0_4854_98a2_cbc699d3cdd8.slice/crio-7a23969a5842a471254cb1043afc017ad814a0aa54376a93a8f397dc04308fc8 WatchSource:0}: Error finding container 7a23969a5842a471254cb1043afc017ad814a0aa54376a93a8f397dc04308fc8: Status 404 returned error can't find the container with id 7a23969a5842a471254cb1043afc017ad814a0aa54376a93a8f397dc04308fc8
Sep 11 16:28:59.477625 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:59.477532    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz" event={"ID":"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8","Type":"ContainerStarted","Data":"0992f194a6d6b3e39d5ca7fd752c13499a8288ce9d41915877d6d7b82f6c3f31"}
Sep 11 16:28:59.477625 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:59.477586    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz" event={"ID":"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8","Type":"ContainerStarted","Data":"7a23969a5842a471254cb1043afc017ad814a0aa54376a93a8f397dc04308fc8"}
Sep 11 16:28:59.477830 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:59.477682    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"
Sep 11 16:28:59.501002 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:28:59.500951    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz" podStartSLOduration=2.043625859 podStartE2EDuration="2.500934301s" totalImagesPullingTime="457.308442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:28:59.499001466 +0000 UTC m=+1599.344645024" watchObservedRunningTime="2026-09-11 16:28:59.500934301 +0000 UTC m=+1599.346577890"
Sep 11 16:29:05.488022 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:05.487992    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"
Sep 11 16:29:30.261586 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.261546    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88"]
Sep 11 16:29:30.267448 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.267420    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88"
Sep 11 16:29:30.272983 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.272958    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2eae3e-dockercfg-zdz62\""
Sep 11 16:29:30.273127 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.272961    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2eae3e\""
Sep 11 16:29:30.302894 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.302866    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88"]
Sep 11 16:29:30.370910 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.370877    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90b041b5-ada9-442c-9e74-d865a4bff30a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88\" (UID: \"90b041b5-ada9-442c-9e74-d865a4bff30a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88"
Sep 11 16:29:30.371124 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.370981    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxrz5\" (UniqueName: \"kubernetes.io/projected/90b041b5-ada9-442c-9e74-d865a4bff30a-kube-api-access-vxrz5\") pod \"payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88\" (UID: \"90b041b5-ada9-442c-9e74-d865a4bff30a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88"
Sep 11 16:29:30.471450 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.471410    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90b041b5-ada9-442c-9e74-d865a4bff30a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88\" (UID: \"90b041b5-ada9-442c-9e74-d865a4bff30a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88"
Sep 11 16:29:30.471656 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.471512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxrz5\" (UniqueName: \"kubernetes.io/projected/90b041b5-ada9-442c-9e74-d865a4bff30a-kube-api-access-vxrz5\") pod \"payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88\" (UID: \"90b041b5-ada9-442c-9e74-d865a4bff30a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88"
Sep 11 16:29:30.472226 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.472201    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90b041b5-ada9-442c-9e74-d865a4bff30a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88\" (UID: \"90b041b5-ada9-442c-9e74-d865a4bff30a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88"
Sep 11 16:29:30.501851 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.501822    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxrz5\" (UniqueName: \"kubernetes.io/projected/90b041b5-ada9-442c-9e74-d865a4bff30a-kube-api-access-vxrz5\") pod \"payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88\" (UID: \"90b041b5-ada9-442c-9e74-d865a4bff30a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88"
Sep 11 16:29:30.533589 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.533510    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc"]
Sep 11 16:29:30.537131 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.537112    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc"
Sep 11 16:29:30.548314 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:29:30.548286    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc\" is forbidden: User \"system:node:ip-10-0-139-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-140.ec2.internal' and this object" podUID="4b0df127-ec55-4412-8e80-579ffb7a0cc6" pod="openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc"
Sep 11 16:29:30.572809 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.572772    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0df127-ec55-4412-8e80-579ffb7a0cc6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc\" (UID: \"4b0df127-ec55-4412-8e80-579ffb7a0cc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc"
Sep 11 16:29:30.572942 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.572910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfcg5\" (UniqueName: \"kubernetes.io/projected/4b0df127-ec55-4412-8e80-579ffb7a0cc6-kube-api-access-qfcg5\") pod \"payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc\" (UID: \"4b0df127-ec55-4412-8e80-579ffb7a0cc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc"
Sep 11 16:29:30.577710 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.577693    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88"
Sep 11 16:29:30.584710 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.584682    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc"]
Sep 11 16:29:30.673401 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.673369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0df127-ec55-4412-8e80-579ffb7a0cc6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc\" (UID: \"4b0df127-ec55-4412-8e80-579ffb7a0cc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc"
Sep 11 16:29:30.673555 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.673465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfcg5\" (UniqueName: \"kubernetes.io/projected/4b0df127-ec55-4412-8e80-579ffb7a0cc6-kube-api-access-qfcg5\") pod \"payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc\" (UID: \"4b0df127-ec55-4412-8e80-579ffb7a0cc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc"
Sep 11 16:29:30.674124 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.674106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0df127-ec55-4412-8e80-579ffb7a0cc6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc\" (UID: \"4b0df127-ec55-4412-8e80-579ffb7a0cc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc"
Sep 11 16:29:30.693554 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.693524    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfcg5\" (UniqueName: \"kubernetes.io/projected/4b0df127-ec55-4412-8e80-579ffb7a0cc6-kube-api-access-qfcg5\") pod \"payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc\" (UID: \"4b0df127-ec55-4412-8e80-579ffb7a0cc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc"
Sep 11 16:29:30.847967 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.847940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc"
Sep 11 16:29:30.884688 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:30.884651    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88"]
Sep 11 16:29:30.893805 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:29:30.893772    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b041b5_ada9_442c_9e74_d865a4bff30a.slice/crio-4cc1f7565ce59be0005a799dfb2c357a345ddb475a020394446afe045b230dba WatchSource:0}: Error finding container 4cc1f7565ce59be0005a799dfb2c357a345ddb475a020394446afe045b230dba: Status 404 returned error can't find the container with id 4cc1f7565ce59be0005a799dfb2c357a345ddb475a020394446afe045b230dba
Sep 11 16:29:31.143232 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:31.143205    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc"]
Sep 11 16:29:31.144841 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:29:31.144818    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0df127_ec55_4412_8e80_579ffb7a0cc6.slice/crio-5ab4705cbbd04eb7fdaee68d4044d9e20ce19177e5a3ef85f66d75db206eca5e WatchSource:0}: Error finding container 5ab4705cbbd04eb7fdaee68d4044d9e20ce19177e5a3ef85f66d75db206eca5e: Status 404 returned error can't find the container with id 5ab4705cbbd04eb7fdaee68d4044d9e20ce19177e5a3ef85f66d75db206eca5e
Sep 11 16:29:31.647025 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:31.646985    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88" event={"ID":"90b041b5-ada9-442c-9e74-d865a4bff30a","Type":"ContainerStarted","Data":"a98e5e1f2fbd825c534a592ad3e0e456bfb8a4c806b5c0d408867d513c4c9f1a"}
Sep 11 16:29:31.647519 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:31.647075    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88"
Sep 11 16:29:31.647519 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:31.647097    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88" event={"ID":"90b041b5-ada9-442c-9e74-d865a4bff30a","Type":"ContainerStarted","Data":"4cc1f7565ce59be0005a799dfb2c357a345ddb475a020394446afe045b230dba"}
Sep 11 16:29:31.648625 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:31.648593    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc" event={"ID":"4b0df127-ec55-4412-8e80-579ffb7a0cc6","Type":"ContainerStarted","Data":"04df46ea6a24de834179b87cb79178ff6d8bbd8806c9ed572cc998dac38d6085"}
Sep 11 16:29:31.648761 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:31.648635    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc" event={"ID":"4b0df127-ec55-4412-8e80-579ffb7a0cc6","Type":"ContainerStarted","Data":"5ab4705cbbd04eb7fdaee68d4044d9e20ce19177e5a3ef85f66d75db206eca5e"}
Sep 11 16:29:31.648761 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:31.648663    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc"
Sep 11 16:29:31.703606 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:31.703554    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88" podStartSLOduration=1.703537538 podStartE2EDuration="1.703537538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:29:31.700190813 +0000 UTC m=+1631.545834367" watchObservedRunningTime="2026-09-11 16:29:31.703537538 +0000 UTC m=+1631.549181092"
Sep 11 16:29:42.656896 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:42.656865    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc"
Sep 11 16:29:42.657356 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:42.656933    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88"
Sep 11 16:29:42.690617 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:29:42.690564    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc" podStartSLOduration=12.690548923 podStartE2EDuration="12.690548923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:29:31.731607317 +0000 UTC m=+1631.577250873" watchObservedRunningTime="2026-09-11 16:29:42.690548923 +0000 UTC m=+1642.536192478"
Sep 11 16:30:00.187562 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:30:00.187527    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9bac29-29819070-w7l6c"]
Sep 11 16:30:00.191387 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:30:00.191364    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9bac29-29819070-w7l6c"
Sep 11 16:30:00.194318 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:30:00.194282    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-ipp-a-9bac29-29819070-w7l6c\" is forbidden: User \"system:node:ip-10-0-139-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-139-140.ec2.internal' and this object" podUID="c74deba5-4afd-4e18-a99d-83f25aa9a7dd" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9bac29-29819070-w7l6c"
Sep 11 16:30:00.194442 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:30:00.194400    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-api-cleanup-dockercfg-t9ktk\" is forbidden: User \"system:node:ip-10-0-139-140.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-139-140.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t9ktk\"" type="*v1.Secret"
Sep 11 16:30:00.215551 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:30:00.215516    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9bac29-29819070-w7l6c"]
Sep 11 16:30:00.753413 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:30:00.753381    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc_4b0df127-ec55-4412-8e80-579ffb7a0cc6/payload-processing/0.log"
Sep 11 16:30:01.203083 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:30:01.203018    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9bac29-29819070-w7l6c" secret="" err="failed to sync secret cache: timed out waiting for the condition"
Sep 11 16:30:01.203480 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:30:01.203136    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9bac29-29819070-w7l6c"
Sep 11 16:30:01.322307 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:30:01.322272    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t9ktk\""
Sep 11 16:30:01.475309 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:30:01.475215    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9bac29-29819070-w7l6c"]
Sep 11 16:30:01.478814 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:30:01.478787    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74deba5_4afd_4e18_a99d_83f25aa9a7dd.slice/crio-1a15be80e8ab99b8085a51ad107c84898a38e7eda172d4cc5223043c629edeb0 WatchSource:0}: Error finding container 1a15be80e8ab99b8085a51ad107c84898a38e7eda172d4cc5223043c629edeb0: Status 404 returned error can't find the container with id 1a15be80e8ab99b8085a51ad107c84898a38e7eda172d4cc5223043c629edeb0
Sep 11 16:30:01.812480 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:30:01.812391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9bac29-29819070-w7l6c" event={"ID":"c74deba5-4afd-4e18-a99d-83f25aa9a7dd","Type":"ContainerStarted","Data":"1a15be80e8ab99b8085a51ad107c84898a38e7eda172d4cc5223043c629edeb0"}
Sep 11 16:30:05.836871 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:30:05.836771    2577 generic.go:356] "Generic (PLEG): container finished" podID="c74deba5-4afd-4e18-a99d-83f25aa9a7dd" containerID="4164c464234dc0c3465b4753c06b47ec6378f533fb08b787705e8d1d63eeae68" exitCode=0
Sep 11 16:30:05.837304 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:30:05.836862    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9bac29-29819070-w7l6c" event={"ID":"c74deba5-4afd-4e18-a99d-83f25aa9a7dd","Type":"ContainerDied","Data":"4164c464234dc0c3465b4753c06b47ec6378f533fb08b787705e8d1d63eeae68"}
Sep 11 16:30:06.976226 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:30:06.976199    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9bac29-29819070-w7l6c"
Sep 11 16:30:07.850452 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:30:07.850427    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9bac29-29819070-w7l6c"
Sep 11 16:30:07.850608 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:30:07.850427    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9bac29-29819070-w7l6c" event={"ID":"c74deba5-4afd-4e18-a99d-83f25aa9a7dd","Type":"ContainerDied","Data":"1a15be80e8ab99b8085a51ad107c84898a38e7eda172d4cc5223043c629edeb0"}
Sep 11 16:30:07.850608 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:30:07.850553    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a15be80e8ab99b8085a51ad107c84898a38e7eda172d4cc5223043c629edeb0"
Sep 11 16:30:46.780822 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:30:46.780742    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc_4b0df127-ec55-4412-8e80-579ffb7a0cc6/payload-processing/0.log"
Sep 11 16:31:14.580482 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:14.580443    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm"]
Sep 11 16:31:14.580935 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:14.580852    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c74deba5-4afd-4e18-a99d-83f25aa9a7dd" containerName="cleanup"
Sep 11 16:31:14.580935 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:14.580864    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c74deba5-4afd-4e18-a99d-83f25aa9a7dd" containerName="cleanup"
Sep 11 16:31:14.580935 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:14.580934    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c74deba5-4afd-4e18-a99d-83f25aa9a7dd" containerName="cleanup"
Sep 11 16:31:14.584084 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:14.584066    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm"
Sep 11 16:31:14.586990 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:14.586969    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-4b91c4\""
Sep 11 16:31:14.587150 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:14.587107    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-4b91c4-dockercfg-nksvm\""
Sep 11 16:31:14.595013 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:14.594986    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm"]
Sep 11 16:31:14.720423 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:14.720386    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20cce86b-af5b-46ba-a3dd-f38f288f83fe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm\" (UID: \"20cce86b-af5b-46ba-a3dd-f38f288f83fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm"
Sep 11 16:31:14.720423 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:14.720423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgqxm\" (UniqueName: \"kubernetes.io/projected/20cce86b-af5b-46ba-a3dd-f38f288f83fe-kube-api-access-hgqxm\") pod \"payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm\" (UID: \"20cce86b-af5b-46ba-a3dd-f38f288f83fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm"
Sep 11 16:31:14.821155 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:14.821105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20cce86b-af5b-46ba-a3dd-f38f288f83fe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm\" (UID: \"20cce86b-af5b-46ba-a3dd-f38f288f83fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm"
Sep 11 16:31:14.821155 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:14.821161    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgqxm\" (UniqueName: \"kubernetes.io/projected/20cce86b-af5b-46ba-a3dd-f38f288f83fe-kube-api-access-hgqxm\") pod \"payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm\" (UID: \"20cce86b-af5b-46ba-a3dd-f38f288f83fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm"
Sep 11 16:31:14.821882 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:14.821856    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20cce86b-af5b-46ba-a3dd-f38f288f83fe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm\" (UID: \"20cce86b-af5b-46ba-a3dd-f38f288f83fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm"
Sep 11 16:31:14.830397 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:14.830371    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgqxm\" (UniqueName: \"kubernetes.io/projected/20cce86b-af5b-46ba-a3dd-f38f288f83fe-kube-api-access-hgqxm\") pod \"payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm\" (UID: \"20cce86b-af5b-46ba-a3dd-f38f288f83fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm"
Sep 11 16:31:14.896203 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:14.896163    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm"
Sep 11 16:31:15.172263 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:15.172238    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm"]
Sep 11 16:31:15.174934 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:31:15.174892    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20cce86b_af5b_46ba_a3dd_f38f288f83fe.slice/crio-7c36df945de3f6e6312f1de93664d904044a44644b7796f9bf704ef6357c318e WatchSource:0}: Error finding container 7c36df945de3f6e6312f1de93664d904044a44644b7796f9bf704ef6357c318e: Status 404 returned error can't find the container with id 7c36df945de3f6e6312f1de93664d904044a44644b7796f9bf704ef6357c318e
Sep 11 16:31:15.194556 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:15.194527    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm" event={"ID":"20cce86b-af5b-46ba-a3dd-f38f288f83fe","Type":"ContainerStarted","Data":"7c36df945de3f6e6312f1de93664d904044a44644b7796f9bf704ef6357c318e"}
Sep 11 16:31:16.202249 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:16.202215    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm" event={"ID":"20cce86b-af5b-46ba-a3dd-f38f288f83fe","Type":"ContainerStarted","Data":"23032b23c47f21133dc8c7f477ef3c323a115595177f52fb3586d7de011727e1"}
Sep 11 16:31:16.202647 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:16.202350    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm"
Sep 11 16:31:16.221332 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:16.221282    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm" podStartSLOduration=2.221265584 podStartE2EDuration="2.221265584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:31:16.219860349 +0000 UTC m=+1736.065503918" watchObservedRunningTime="2026-09-11 16:31:16.221265584 +0000 UTC m=+1736.066909139"
Sep 11 16:31:27.209288 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:27.209248    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm"
Sep 11 16:31:53.169055 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:53.169003    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm"]
Sep 11 16:31:53.169566 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:53.169287    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm" podUID="20cce86b-af5b-46ba-a3dd-f38f288f83fe" containerName="payload-processing" containerID="cri-o://23032b23c47f21133dc8c7f477ef3c323a115595177f52fb3586d7de011727e1" gracePeriod=30
Sep 11 16:31:53.403363 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:53.403332    2577 generic.go:356] "Generic (PLEG): container finished" podID="20cce86b-af5b-46ba-a3dd-f38f288f83fe" containerID="23032b23c47f21133dc8c7f477ef3c323a115595177f52fb3586d7de011727e1" exitCode=0
Sep 11 16:31:53.403552 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:53.403403    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm" event={"ID":"20cce86b-af5b-46ba-a3dd-f38f288f83fe","Type":"ContainerDied","Data":"23032b23c47f21133dc8c7f477ef3c323a115595177f52fb3586d7de011727e1"}
Sep 11 16:31:53.433017 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:53.432954    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm"
Sep 11 16:31:53.456583 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:53.456549    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20cce86b-af5b-46ba-a3dd-f38f288f83fe-plugins-config-volume\") pod \"20cce86b-af5b-46ba-a3dd-f38f288f83fe\" (UID: \"20cce86b-af5b-46ba-a3dd-f38f288f83fe\") "
Sep 11 16:31:53.456778 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:53.456608    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgqxm\" (UniqueName: \"kubernetes.io/projected/20cce86b-af5b-46ba-a3dd-f38f288f83fe-kube-api-access-hgqxm\") pod \"20cce86b-af5b-46ba-a3dd-f38f288f83fe\" (UID: \"20cce86b-af5b-46ba-a3dd-f38f288f83fe\") "
Sep 11 16:31:53.457053 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:53.456988    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20cce86b-af5b-46ba-a3dd-f38f288f83fe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "20cce86b-af5b-46ba-a3dd-f38f288f83fe" (UID: "20cce86b-af5b-46ba-a3dd-f38f288f83fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:31:53.459561 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:53.459520    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20cce86b-af5b-46ba-a3dd-f38f288f83fe-kube-api-access-hgqxm" (OuterVolumeSpecName: "kube-api-access-hgqxm") pod "20cce86b-af5b-46ba-a3dd-f38f288f83fe" (UID: "20cce86b-af5b-46ba-a3dd-f38f288f83fe"). InnerVolumeSpecName "kube-api-access-hgqxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:31:53.558204 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:53.558167    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20cce86b-af5b-46ba-a3dd-f38f288f83fe-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:31:53.558204 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:53.558199    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgqxm\" (UniqueName: \"kubernetes.io/projected/20cce86b-af5b-46ba-a3dd-f38f288f83fe-kube-api-access-hgqxm\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:31:54.410575 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:54.410542    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm"
Sep 11 16:31:54.410575 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:54.410572    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm" event={"ID":"20cce86b-af5b-46ba-a3dd-f38f288f83fe","Type":"ContainerDied","Data":"7c36df945de3f6e6312f1de93664d904044a44644b7796f9bf704ef6357c318e"}
Sep 11 16:31:54.411110 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:54.410631    2577 scope.go:117] "RemoveContainer" containerID="23032b23c47f21133dc8c7f477ef3c323a115595177f52fb3586d7de011727e1"
Sep 11 16:31:54.434516 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:54.434488    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm"]
Sep 11 16:31:54.442323 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:54.442300    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-4b91c4-876c7c977-tf6dm"]
Sep 11 16:31:54.633838 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:31:54.633808    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20cce86b-af5b-46ba-a3dd-f38f288f83fe" path="/var/lib/kubelet/pods/20cce86b-af5b-46ba-a3dd-f38f288f83fe/volumes"
Sep 11 16:32:19.876279 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:19.876244    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"]
Sep 11 16:32:19.876776 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:19.876470    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz" podUID="d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8" containerName="maas-api" containerID="cri-o://0992f194a6d6b3e39d5ca7fd752c13499a8288ce9d41915877d6d7b82f6c3f31" gracePeriod=30
Sep 11 16:32:19.918559 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:19.918529    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9bac29-29819070-w7l6c"]
Sep 11 16:32:19.920324 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:19.920297    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9bac29-29819070-w7l6c"]
Sep 11 16:32:20.140273 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.140210    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"
Sep 11 16:32:20.199744 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.199712    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-openshift-service-ca\") pod \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\" (UID: \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\") "
Sep 11 16:32:20.199744 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.199750    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-maas-api-tls\") pod \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\" (UID: \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\") "
Sep 11 16:32:20.199951 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.199802    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-metrics-certs\") pod \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\" (UID: \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\") "
Sep 11 16:32:20.199951 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.199894    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pncpp\" (UniqueName: \"kubernetes.io/projected/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-kube-api-access-pncpp\") pod \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\" (UID: \"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8\") "
Sep 11 16:32:20.200170 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.200143    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8" (UID: "d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:32:20.202136 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.202106    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8" (UID: "d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:32:20.202257 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.202228    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8" (UID: "d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:32:20.202316 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.202291    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-kube-api-access-pncpp" (OuterVolumeSpecName: "kube-api-access-pncpp") pod "d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8" (UID: "d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8"). InnerVolumeSpecName "kube-api-access-pncpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:32:20.301090 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.301007    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-openshift-service-ca\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:32:20.301090 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.301087    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-maas-api-tls\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:32:20.301090 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.301100    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-metrics-certs\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:32:20.301392 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.301113    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pncpp\" (UniqueName: \"kubernetes.io/projected/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8-kube-api-access-pncpp\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:32:20.546714 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.546621    2577 generic.go:356] "Generic (PLEG): container finished" podID="d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8" containerID="0992f194a6d6b3e39d5ca7fd752c13499a8288ce9d41915877d6d7b82f6c3f31" exitCode=0
Sep 11 16:32:20.546714 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.546667    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"
Sep 11 16:32:20.546714 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.546672    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz" event={"ID":"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8","Type":"ContainerDied","Data":"0992f194a6d6b3e39d5ca7fd752c13499a8288ce9d41915877d6d7b82f6c3f31"}
Sep 11 16:32:20.546943 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.546728    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz" event={"ID":"d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8","Type":"ContainerDied","Data":"7a23969a5842a471254cb1043afc017ad814a0aa54376a93a8f397dc04308fc8"}
Sep 11 16:32:20.546943 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.546759    2577 scope.go:117] "RemoveContainer" containerID="0992f194a6d6b3e39d5ca7fd752c13499a8288ce9d41915877d6d7b82f6c3f31"
Sep 11 16:32:20.556685 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.556664    2577 scope.go:117] "RemoveContainer" containerID="0992f194a6d6b3e39d5ca7fd752c13499a8288ce9d41915877d6d7b82f6c3f31"
Sep 11 16:32:20.556968 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:32:20.556947    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0992f194a6d6b3e39d5ca7fd752c13499a8288ce9d41915877d6d7b82f6c3f31\": container with ID starting with 0992f194a6d6b3e39d5ca7fd752c13499a8288ce9d41915877d6d7b82f6c3f31 not found: ID does not exist" containerID="0992f194a6d6b3e39d5ca7fd752c13499a8288ce9d41915877d6d7b82f6c3f31"
Sep 11 16:32:20.557021 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.556977    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0992f194a6d6b3e39d5ca7fd752c13499a8288ce9d41915877d6d7b82f6c3f31"} err="failed to get container status \"0992f194a6d6b3e39d5ca7fd752c13499a8288ce9d41915877d6d7b82f6c3f31\": rpc error: code = NotFound desc = could not find container \"0992f194a6d6b3e39d5ca7fd752c13499a8288ce9d41915877d6d7b82f6c3f31\": container with ID starting with 0992f194a6d6b3e39d5ca7fd752c13499a8288ce9d41915877d6d7b82f6c3f31 not found: ID does not exist"
Sep 11 16:32:20.573223 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.573192    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"]
Sep 11 16:32:20.577312 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.577290    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9bac29-74dd8cf847-pdzxz"]
Sep 11 16:32:20.632755 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.632727    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c74deba5-4afd-4e18-a99d-83f25aa9a7dd" path="/var/lib/kubelet/pods/c74deba5-4afd-4e18-a99d-83f25aa9a7dd/volumes"
Sep 11 16:32:20.633102 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.633088    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8" path="/var/lib/kubelet/pods/d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8/volumes"
Sep 11 16:32:20.669618 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.669594    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/1.log"
Sep 11 16:32:20.673026 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.673000    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/1.log"
Sep 11 16:32:20.677130 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.677109    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ng2ww_07d57c04-0697-419b-9d3e-ca11761da85c/ovn-acl-logging/0.log"
Sep 11 16:32:20.680089 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:32:20.680070    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ng2ww_07d57c04-0697-419b-9d3e-ca11761da85c/ovn-acl-logging/0.log"
Sep 11 16:33:22.425344 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.425305    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc"]
Sep 11 16:33:22.427918 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.425526    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc" podUID="4b0df127-ec55-4412-8e80-579ffb7a0cc6" containerName="payload-processing" containerID="cri-o://04df46ea6a24de834179b87cb79178ff6d8bbd8806c9ed572cc998dac38d6085" gracePeriod=30
Sep 11 16:33:22.443188 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.443155    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88"]
Sep 11 16:33:22.443466 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.443412    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88" podUID="90b041b5-ada9-442c-9e74-d865a4bff30a" containerName="payload-pre-processing" containerID="cri-o://a98e5e1f2fbd825c534a592ad3e0e456bfb8a4c806b5c0d408867d513c4c9f1a" gracePeriod=30
Sep 11 16:33:22.703202 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.703173    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc"
Sep 11 16:33:22.734552 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.734524    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88"
Sep 11 16:33:22.828312 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.828278    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxrz5\" (UniqueName: \"kubernetes.io/projected/90b041b5-ada9-442c-9e74-d865a4bff30a-kube-api-access-vxrz5\") pod \"90b041b5-ada9-442c-9e74-d865a4bff30a\" (UID: \"90b041b5-ada9-442c-9e74-d865a4bff30a\") "
Sep 11 16:33:22.828489 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.828402    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90b041b5-ada9-442c-9e74-d865a4bff30a-plugins-config-volume\") pod \"90b041b5-ada9-442c-9e74-d865a4bff30a\" (UID: \"90b041b5-ada9-442c-9e74-d865a4bff30a\") "
Sep 11 16:33:22.828489 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.828429    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0df127-ec55-4412-8e80-579ffb7a0cc6-plugins-config-volume\") pod \"4b0df127-ec55-4412-8e80-579ffb7a0cc6\" (UID: \"4b0df127-ec55-4412-8e80-579ffb7a0cc6\") "
Sep 11 16:33:22.828489 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.828453    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfcg5\" (UniqueName: \"kubernetes.io/projected/4b0df127-ec55-4412-8e80-579ffb7a0cc6-kube-api-access-qfcg5\") pod \"4b0df127-ec55-4412-8e80-579ffb7a0cc6\" (UID: \"4b0df127-ec55-4412-8e80-579ffb7a0cc6\") "
Sep 11 16:33:22.828801 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.828771    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b0df127-ec55-4412-8e80-579ffb7a0cc6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b0df127-ec55-4412-8e80-579ffb7a0cc6" (UID: "4b0df127-ec55-4412-8e80-579ffb7a0cc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:33:22.828867 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.828834    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b041b5-ada9-442c-9e74-d865a4bff30a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "90b041b5-ada9-442c-9e74-d865a4bff30a" (UID: "90b041b5-ada9-442c-9e74-d865a4bff30a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:33:22.830774 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.830730    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b041b5-ada9-442c-9e74-d865a4bff30a-kube-api-access-vxrz5" (OuterVolumeSpecName: "kube-api-access-vxrz5") pod "90b041b5-ada9-442c-9e74-d865a4bff30a" (UID: "90b041b5-ada9-442c-9e74-d865a4bff30a"). InnerVolumeSpecName "kube-api-access-vxrz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:33:22.830915 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.830864    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b0df127-ec55-4412-8e80-579ffb7a0cc6-kube-api-access-qfcg5" (OuterVolumeSpecName: "kube-api-access-qfcg5") pod "4b0df127-ec55-4412-8e80-579ffb7a0cc6" (UID: "4b0df127-ec55-4412-8e80-579ffb7a0cc6"). InnerVolumeSpecName "kube-api-access-qfcg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:33:22.860955 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.860921    2577 generic.go:356] "Generic (PLEG): container finished" podID="90b041b5-ada9-442c-9e74-d865a4bff30a" containerID="a98e5e1f2fbd825c534a592ad3e0e456bfb8a4c806b5c0d408867d513c4c9f1a" exitCode=0
Sep 11 16:33:22.861187 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.860992    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88"
Sep 11 16:33:22.861187 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.861013    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88" event={"ID":"90b041b5-ada9-442c-9e74-d865a4bff30a","Type":"ContainerDied","Data":"a98e5e1f2fbd825c534a592ad3e0e456bfb8a4c806b5c0d408867d513c4c9f1a"}
Sep 11 16:33:22.861187 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.861131    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88" event={"ID":"90b041b5-ada9-442c-9e74-d865a4bff30a","Type":"ContainerDied","Data":"4cc1f7565ce59be0005a799dfb2c357a345ddb475a020394446afe045b230dba"}
Sep 11 16:33:22.861187 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.861152    2577 scope.go:117] "RemoveContainer" containerID="a98e5e1f2fbd825c534a592ad3e0e456bfb8a4c806b5c0d408867d513c4c9f1a"
Sep 11 16:33:22.862642 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.862616    2577 generic.go:356] "Generic (PLEG): container finished" podID="4b0df127-ec55-4412-8e80-579ffb7a0cc6" containerID="04df46ea6a24de834179b87cb79178ff6d8bbd8806c9ed572cc998dac38d6085" exitCode=0
Sep 11 16:33:22.862759 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.862702    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc" event={"ID":"4b0df127-ec55-4412-8e80-579ffb7a0cc6","Type":"ContainerDied","Data":"04df46ea6a24de834179b87cb79178ff6d8bbd8806c9ed572cc998dac38d6085"}
Sep 11 16:33:22.862759 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.862748    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc" event={"ID":"4b0df127-ec55-4412-8e80-579ffb7a0cc6","Type":"ContainerDied","Data":"5ab4705cbbd04eb7fdaee68d4044d9e20ce19177e5a3ef85f66d75db206eca5e"}
Sep 11 16:33:22.862881 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.862835    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc"
Sep 11 16:33:22.870956 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.870923    2577 scope.go:117] "RemoveContainer" containerID="a98e5e1f2fbd825c534a592ad3e0e456bfb8a4c806b5c0d408867d513c4c9f1a"
Sep 11 16:33:22.871519 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:33:22.871326    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a98e5e1f2fbd825c534a592ad3e0e456bfb8a4c806b5c0d408867d513c4c9f1a\": container with ID starting with a98e5e1f2fbd825c534a592ad3e0e456bfb8a4c806b5c0d408867d513c4c9f1a not found: ID does not exist" containerID="a98e5e1f2fbd825c534a592ad3e0e456bfb8a4c806b5c0d408867d513c4c9f1a"
Sep 11 16:33:22.871519 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.871364    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a98e5e1f2fbd825c534a592ad3e0e456bfb8a4c806b5c0d408867d513c4c9f1a"} err="failed to get container status \"a98e5e1f2fbd825c534a592ad3e0e456bfb8a4c806b5c0d408867d513c4c9f1a\": rpc error: code = NotFound desc = could not find container \"a98e5e1f2fbd825c534a592ad3e0e456bfb8a4c806b5c0d408867d513c4c9f1a\": container with ID starting with a98e5e1f2fbd825c534a592ad3e0e456bfb8a4c806b5c0d408867d513c4c9f1a not found: ID does not exist"
Sep 11 16:33:22.871519 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.871388    2577 scope.go:117] "RemoveContainer" containerID="04df46ea6a24de834179b87cb79178ff6d8bbd8806c9ed572cc998dac38d6085"
Sep 11 16:33:22.881169 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.881150    2577 scope.go:117] "RemoveContainer" containerID="04df46ea6a24de834179b87cb79178ff6d8bbd8806c9ed572cc998dac38d6085"
Sep 11 16:33:22.881522 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:33:22.881495    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04df46ea6a24de834179b87cb79178ff6d8bbd8806c9ed572cc998dac38d6085\": container with ID starting with 04df46ea6a24de834179b87cb79178ff6d8bbd8806c9ed572cc998dac38d6085 not found: ID does not exist" containerID="04df46ea6a24de834179b87cb79178ff6d8bbd8806c9ed572cc998dac38d6085"
Sep 11 16:33:22.881630 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.881528    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04df46ea6a24de834179b87cb79178ff6d8bbd8806c9ed572cc998dac38d6085"} err="failed to get container status \"04df46ea6a24de834179b87cb79178ff6d8bbd8806c9ed572cc998dac38d6085\": rpc error: code = NotFound desc = could not find container \"04df46ea6a24de834179b87cb79178ff6d8bbd8806c9ed572cc998dac38d6085\": container with ID starting with 04df46ea6a24de834179b87cb79178ff6d8bbd8806c9ed572cc998dac38d6085 not found: ID does not exist"
Sep 11 16:33:22.890841 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.890810    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc"]
Sep 11 16:33:22.894621 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.894592    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc"]
Sep 11 16:33:22.911995 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.911958    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88"]
Sep 11 16:33:22.915930 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.915902    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88"]
Sep 11 16:33:22.929387 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.929343    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxrz5\" (UniqueName: \"kubernetes.io/projected/90b041b5-ada9-442c-9e74-d865a4bff30a-kube-api-access-vxrz5\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:33:22.929387 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.929388    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90b041b5-ada9-442c-9e74-d865a4bff30a-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:33:22.929627 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.929407    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0df127-ec55-4412-8e80-579ffb7a0cc6-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:33:22.929627 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:22.929417    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfcg5\" (UniqueName: \"kubernetes.io/projected/4b0df127-ec55-4412-8e80-579ffb7a0cc6-kube-api-access-qfcg5\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:33:23.654893 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:23.654847    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-2eae3e-ffc99b85c-4t4sc" podUID="4b0df127-ec55-4412-8e80-579ffb7a0cc6" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.75:9004: i/o timeout"
Sep 11 16:33:23.654893 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:23.654865    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2eae3e-bb6795458-jdl88" podUID="90b041b5-ada9-442c-9e74-d865a4bff30a" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.74:9004: i/o timeout"
Sep 11 16:33:24.633154 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:24.633124    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b0df127-ec55-4412-8e80-579ffb7a0cc6" path="/var/lib/kubelet/pods/4b0df127-ec55-4412-8e80-579ffb7a0cc6/volumes"
Sep 11 16:33:24.633506 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:24.633492    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90b041b5-ada9-442c-9e74-d865a4bff30a" path="/var/lib/kubelet/pods/90b041b5-ada9-442c-9e74-d865a4bff30a/volumes"
Sep 11 16:33:42.067851 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.067762    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6"]
Sep 11 16:33:42.068344 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.068026    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6" podUID="63051b3a-b943-4742-9ee1-a48d2ae7f28a" containerName="payload-processing" containerID="cri-o://5d8bfa8437d4dcd315a07b5e78edbf270b2262ae2609aec0c6ea3a180de04b02" gracePeriod=30
Sep 11 16:33:42.083582 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.083551    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9"]
Sep 11 16:33:42.083798 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.083760    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9" podUID="4311d371-c6e8-4d66-8bce-fbea05486610" containerName="payload-pre-processing" containerID="cri-o://f9ef5700e890009f9cb3ae755aad4d94ce069dcbda6944eb1725c1b65f62e9db" gracePeriod=30
Sep 11 16:33:42.329853 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.329786    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6"
Sep 11 16:33:42.365820 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.365791    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9"
Sep 11 16:33:42.410348 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.410315    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63051b3a-b943-4742-9ee1-a48d2ae7f28a-plugins-config-volume\") pod \"63051b3a-b943-4742-9ee1-a48d2ae7f28a\" (UID: \"63051b3a-b943-4742-9ee1-a48d2ae7f28a\") "
Sep 11 16:33:42.410523 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.410384    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djn4n\" (UniqueName: \"kubernetes.io/projected/63051b3a-b943-4742-9ee1-a48d2ae7f28a-kube-api-access-djn4n\") pod \"63051b3a-b943-4742-9ee1-a48d2ae7f28a\" (UID: \"63051b3a-b943-4742-9ee1-a48d2ae7f28a\") "
Sep 11 16:33:42.410798 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.410763    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63051b3a-b943-4742-9ee1-a48d2ae7f28a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "63051b3a-b943-4742-9ee1-a48d2ae7f28a" (UID: "63051b3a-b943-4742-9ee1-a48d2ae7f28a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:33:42.412704 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.412672    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63051b3a-b943-4742-9ee1-a48d2ae7f28a-kube-api-access-djn4n" (OuterVolumeSpecName: "kube-api-access-djn4n") pod "63051b3a-b943-4742-9ee1-a48d2ae7f28a" (UID: "63051b3a-b943-4742-9ee1-a48d2ae7f28a"). InnerVolumeSpecName "kube-api-access-djn4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:33:42.511459 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.511425    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bngt\" (UniqueName: \"kubernetes.io/projected/4311d371-c6e8-4d66-8bce-fbea05486610-kube-api-access-5bngt\") pod \"4311d371-c6e8-4d66-8bce-fbea05486610\" (UID: \"4311d371-c6e8-4d66-8bce-fbea05486610\") "
Sep 11 16:33:42.511649 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.511513    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4311d371-c6e8-4d66-8bce-fbea05486610-plugins-config-volume\") pod \"4311d371-c6e8-4d66-8bce-fbea05486610\" (UID: \"4311d371-c6e8-4d66-8bce-fbea05486610\") "
Sep 11 16:33:42.511721 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.511703    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63051b3a-b943-4742-9ee1-a48d2ae7f28a-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:33:42.511761 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.511727    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djn4n\" (UniqueName: \"kubernetes.io/projected/63051b3a-b943-4742-9ee1-a48d2ae7f28a-kube-api-access-djn4n\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:33:42.511910 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.511890    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4311d371-c6e8-4d66-8bce-fbea05486610-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4311d371-c6e8-4d66-8bce-fbea05486610" (UID: "4311d371-c6e8-4d66-8bce-fbea05486610"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:33:42.513748 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.513724    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4311d371-c6e8-4d66-8bce-fbea05486610-kube-api-access-5bngt" (OuterVolumeSpecName: "kube-api-access-5bngt") pod "4311d371-c6e8-4d66-8bce-fbea05486610" (UID: "4311d371-c6e8-4d66-8bce-fbea05486610"). InnerVolumeSpecName "kube-api-access-5bngt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:33:42.612656 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.612604    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4311d371-c6e8-4d66-8bce-fbea05486610-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:33:42.612656 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.612645    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bngt\" (UniqueName: \"kubernetes.io/projected/4311d371-c6e8-4d66-8bce-fbea05486610-kube-api-access-5bngt\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:33:42.973300 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.973209    2577 generic.go:356] "Generic (PLEG): container finished" podID="63051b3a-b943-4742-9ee1-a48d2ae7f28a" containerID="5d8bfa8437d4dcd315a07b5e78edbf270b2262ae2609aec0c6ea3a180de04b02" exitCode=0
Sep 11 16:33:42.973300 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.973258    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6"
Sep 11 16:33:42.973501 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.973299    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6" event={"ID":"63051b3a-b943-4742-9ee1-a48d2ae7f28a","Type":"ContainerDied","Data":"5d8bfa8437d4dcd315a07b5e78edbf270b2262ae2609aec0c6ea3a180de04b02"}
Sep 11 16:33:42.973501 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.973356    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6" event={"ID":"63051b3a-b943-4742-9ee1-a48d2ae7f28a","Type":"ContainerDied","Data":"3d00f8d1c4cfbc1bacfce39afa88e11bb6d1fdfcb684fd8bface169f21f6e1bc"}
Sep 11 16:33:42.973501 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.973383    2577 scope.go:117] "RemoveContainer" containerID="5d8bfa8437d4dcd315a07b5e78edbf270b2262ae2609aec0c6ea3a180de04b02"
Sep 11 16:33:42.974639 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.974580    2577 generic.go:356] "Generic (PLEG): container finished" podID="4311d371-c6e8-4d66-8bce-fbea05486610" containerID="f9ef5700e890009f9cb3ae755aad4d94ce069dcbda6944eb1725c1b65f62e9db" exitCode=0
Sep 11 16:33:42.974639 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.974610    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9" event={"ID":"4311d371-c6e8-4d66-8bce-fbea05486610","Type":"ContainerDied","Data":"f9ef5700e890009f9cb3ae755aad4d94ce069dcbda6944eb1725c1b65f62e9db"}
Sep 11 16:33:42.974639 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.974634    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9"
Sep 11 16:33:42.974795 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.974672    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9" event={"ID":"4311d371-c6e8-4d66-8bce-fbea05486610","Type":"ContainerDied","Data":"f1de028a9dcd0ea212b0c6dcb8e96f3cbf8f6fa5c1d5cc07994c038de406fc71"}
Sep 11 16:33:42.982549 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.982534    2577 scope.go:117] "RemoveContainer" containerID="5d8bfa8437d4dcd315a07b5e78edbf270b2262ae2609aec0c6ea3a180de04b02"
Sep 11 16:33:42.982822 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:33:42.982804    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d8bfa8437d4dcd315a07b5e78edbf270b2262ae2609aec0c6ea3a180de04b02\": container with ID starting with 5d8bfa8437d4dcd315a07b5e78edbf270b2262ae2609aec0c6ea3a180de04b02 not found: ID does not exist" containerID="5d8bfa8437d4dcd315a07b5e78edbf270b2262ae2609aec0c6ea3a180de04b02"
Sep 11 16:33:42.982897 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.982828    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d8bfa8437d4dcd315a07b5e78edbf270b2262ae2609aec0c6ea3a180de04b02"} err="failed to get container status \"5d8bfa8437d4dcd315a07b5e78edbf270b2262ae2609aec0c6ea3a180de04b02\": rpc error: code = NotFound desc = could not find container \"5d8bfa8437d4dcd315a07b5e78edbf270b2262ae2609aec0c6ea3a180de04b02\": container with ID starting with 5d8bfa8437d4dcd315a07b5e78edbf270b2262ae2609aec0c6ea3a180de04b02 not found: ID does not exist"
Sep 11 16:33:42.982897 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.982844    2577 scope.go:117] "RemoveContainer" containerID="f9ef5700e890009f9cb3ae755aad4d94ce069dcbda6944eb1725c1b65f62e9db"
Sep 11 16:33:42.990910 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.990892    2577 scope.go:117] "RemoveContainer" containerID="f9ef5700e890009f9cb3ae755aad4d94ce069dcbda6944eb1725c1b65f62e9db"
Sep 11 16:33:42.991266 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:33:42.991214    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ef5700e890009f9cb3ae755aad4d94ce069dcbda6944eb1725c1b65f62e9db\": container with ID starting with f9ef5700e890009f9cb3ae755aad4d94ce069dcbda6944eb1725c1b65f62e9db not found: ID does not exist" containerID="f9ef5700e890009f9cb3ae755aad4d94ce069dcbda6944eb1725c1b65f62e9db"
Sep 11 16:33:42.991354 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.991274    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ef5700e890009f9cb3ae755aad4d94ce069dcbda6944eb1725c1b65f62e9db"} err="failed to get container status \"f9ef5700e890009f9cb3ae755aad4d94ce069dcbda6944eb1725c1b65f62e9db\": rpc error: code = NotFound desc = could not find container \"f9ef5700e890009f9cb3ae755aad4d94ce069dcbda6944eb1725c1b65f62e9db\": container with ID starting with f9ef5700e890009f9cb3ae755aad4d94ce069dcbda6944eb1725c1b65f62e9db not found: ID does not exist"
Sep 11 16:33:42.993849 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.993828    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9"]
Sep 11 16:33:42.998723 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:42.998697    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1ec6e6-6b7b8d5b58-hmcf9"]
Sep 11 16:33:43.011250 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:43.011223    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6"]
Sep 11 16:33:43.016581 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:43.016559    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1ec6e6-5c6f49fdd7-7l7d6"]
Sep 11 16:33:44.633769 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:44.633739    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4311d371-c6e8-4d66-8bce-fbea05486610" path="/var/lib/kubelet/pods/4311d371-c6e8-4d66-8bce-fbea05486610/volumes"
Sep 11 16:33:44.634232 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:33:44.634204    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63051b3a-b943-4742-9ee1-a48d2ae7f28a" path="/var/lib/kubelet/pods/63051b3a-b943-4742-9ee1-a48d2ae7f28a/volumes"
Sep 11 16:34:00.520154 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:34:00.520117    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4"]
Sep 11 16:34:00.520683 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:34:00.520358    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4" podUID="6897d110-2a89-4930-92f3-c74204aff896" containerName="payload-pre-processing" containerID="cri-o://40bd62ad586554ce9e48854989897ae7aa14599872e0ce6ea1895ec9b9bb994f" gracePeriod=30
Sep 11 16:34:00.779107 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:34:00.779018    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4"
Sep 11 16:34:00.866279 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:34:00.866243    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6897d110-2a89-4930-92f3-c74204aff896-plugins-config-volume\") pod \"6897d110-2a89-4930-92f3-c74204aff896\" (UID: \"6897d110-2a89-4930-92f3-c74204aff896\") "
Sep 11 16:34:00.866279 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:34:00.866288    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p4pv\" (UniqueName: \"kubernetes.io/projected/6897d110-2a89-4930-92f3-c74204aff896-kube-api-access-5p4pv\") pod \"6897d110-2a89-4930-92f3-c74204aff896\" (UID: \"6897d110-2a89-4930-92f3-c74204aff896\") "
Sep 11 16:34:00.866636 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:34:00.866611    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6897d110-2a89-4930-92f3-c74204aff896-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6897d110-2a89-4930-92f3-c74204aff896" (UID: "6897d110-2a89-4930-92f3-c74204aff896"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:34:00.868549 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:34:00.868528    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6897d110-2a89-4930-92f3-c74204aff896-kube-api-access-5p4pv" (OuterVolumeSpecName: "kube-api-access-5p4pv") pod "6897d110-2a89-4930-92f3-c74204aff896" (UID: "6897d110-2a89-4930-92f3-c74204aff896"). InnerVolumeSpecName "kube-api-access-5p4pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:34:00.967402 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:34:00.967371    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6897d110-2a89-4930-92f3-c74204aff896-plugins-config-volume\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:34:00.967402 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:34:00.967396    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p4pv\" (UniqueName: \"kubernetes.io/projected/6897d110-2a89-4930-92f3-c74204aff896-kube-api-access-5p4pv\") on node \"ip-10-0-139-140.ec2.internal\" DevicePath \"\""
Sep 11 16:34:01.068392 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:34:01.068303    2577 generic.go:356] "Generic (PLEG): container finished" podID="6897d110-2a89-4930-92f3-c74204aff896" containerID="40bd62ad586554ce9e48854989897ae7aa14599872e0ce6ea1895ec9b9bb994f" exitCode=0
Sep 11 16:34:01.068392 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:34:01.068348    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4"
Sep 11 16:34:01.068573 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:34:01.068343    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4" event={"ID":"6897d110-2a89-4930-92f3-c74204aff896","Type":"ContainerDied","Data":"40bd62ad586554ce9e48854989897ae7aa14599872e0ce6ea1895ec9b9bb994f"}
Sep 11 16:34:01.068573 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:34:01.068467    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4" event={"ID":"6897d110-2a89-4930-92f3-c74204aff896","Type":"ContainerDied","Data":"6fb1442a77abf1ae6962d8349d4e85b342980387688200cca034026bc8ae1483"}
Sep 11 16:34:01.068573 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:34:01.068498    2577 scope.go:117] "RemoveContainer" containerID="40bd62ad586554ce9e48854989897ae7aa14599872e0ce6ea1895ec9b9bb994f"
Sep 11 16:34:01.080077 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:34:01.079980    2577 scope.go:117] "RemoveContainer" containerID="40bd62ad586554ce9e48854989897ae7aa14599872e0ce6ea1895ec9b9bb994f"
Sep 11 16:34:01.082714 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:34:01.082686    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40bd62ad586554ce9e48854989897ae7aa14599872e0ce6ea1895ec9b9bb994f\": container with ID starting with 40bd62ad586554ce9e48854989897ae7aa14599872e0ce6ea1895ec9b9bb994f not found: ID does not exist" containerID="40bd62ad586554ce9e48854989897ae7aa14599872e0ce6ea1895ec9b9bb994f"
Sep 11 16:34:01.082836 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:34:01.082723    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40bd62ad586554ce9e48854989897ae7aa14599872e0ce6ea1895ec9b9bb994f"} err="failed to get container status \"40bd62ad586554ce9e48854989897ae7aa14599872e0ce6ea1895ec9b9bb994f\": rpc error: code = NotFound desc = could not find container \"40bd62ad586554ce9e48854989897ae7aa14599872e0ce6ea1895ec9b9bb994f\": container with ID starting with 40bd62ad586554ce9e48854989897ae7aa14599872e0ce6ea1895ec9b9bb994f not found: ID does not exist"
Sep 11 16:34:01.104796 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:34:01.104767    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4"]
Sep 11 16:34:01.112294 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:34:01.112266    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-233d22-567b67558b-htmc4"]
Sep 11 16:34:02.632557 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:34:02.632521    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6897d110-2a89-4930-92f3-c74204aff896" path="/var/lib/kubelet/pods/6897d110-2a89-4930-92f3-c74204aff896/volumes"
Sep 11 16:35:29.386231 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.386189    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z" podUID="b403831a-b82b-486e-9c63-2b7382bb238e" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.23:8081/readyz\": dial tcp 10.134.0.23:8081: connect: connection refused"
Sep 11 16:35:29.507218 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.507188    2577 generic.go:356] "Generic (PLEG): container finished" podID="89912015-3e6c-41e8-8ce3-1af04df3584d" containerID="d247bb25b8df8d65d536890247acef3bf4add5dda37127dd333ba2dcd1547a02" exitCode=1
Sep 11 16:35:29.507372 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.507268    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c" event={"ID":"89912015-3e6c-41e8-8ce3-1af04df3584d","Type":"ContainerDied","Data":"d247bb25b8df8d65d536890247acef3bf4add5dda37127dd333ba2dcd1547a02"}
Sep 11 16:35:29.507647 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.507620    2577 scope.go:117] "RemoveContainer" containerID="d247bb25b8df8d65d536890247acef3bf4add5dda37127dd333ba2dcd1547a02"
Sep 11 16:35:29.508975 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.508951    2577 generic.go:356] "Generic (PLEG): container finished" podID="47a55b39-a458-4484-af8f-07a70f9de757" containerID="4694e8a00e39848e9b8e750976de1c0234980eed356773926d574216a4ff4ae4" exitCode=1
Sep 11 16:35:29.509084 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.508975    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-vncpm" event={"ID":"47a55b39-a458-4484-af8f-07a70f9de757","Type":"ContainerDied","Data":"4694e8a00e39848e9b8e750976de1c0234980eed356773926d574216a4ff4ae4"}
Sep 11 16:35:29.509316 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.509301    2577 scope.go:117] "RemoveContainer" containerID="4694e8a00e39848e9b8e750976de1c0234980eed356773926d574216a4ff4ae4"
Sep 11 16:35:29.509596 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.509581    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:35:29.510956 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.510937    2577 generic.go:356] "Generic (PLEG): container finished" podID="96a83de2-03be-4483-a321-7f71b53a3a1b" containerID="8ed23ec73f42eefb2cd9d28e11bf9dc4f8a5f6a635664b267b7e117a8be0032a" exitCode=0
Sep 11 16:35:29.511174 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.511012    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm" event={"ID":"96a83de2-03be-4483-a321-7f71b53a3a1b","Type":"ContainerDied","Data":"8ed23ec73f42eefb2cd9d28e11bf9dc4f8a5f6a635664b267b7e117a8be0032a"}
Sep 11 16:35:29.511359 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.511343    2577 scope.go:117] "RemoveContainer" containerID="8ed23ec73f42eefb2cd9d28e11bf9dc4f8a5f6a635664b267b7e117a8be0032a"
Sep 11 16:35:29.513241 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.513187    2577 generic.go:356] "Generic (PLEG): container finished" podID="b8929d82-a6e4-4e3c-a9e5-8158a14652bb" containerID="0a680931a955b912451051d8b55a5fbf25f4203294cdfdd8a5fde767d01aba7f" exitCode=1
Sep 11 16:35:29.513313 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.513242    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz" event={"ID":"b8929d82-a6e4-4e3c-a9e5-8158a14652bb","Type":"ContainerDied","Data":"0a680931a955b912451051d8b55a5fbf25f4203294cdfdd8a5fde767d01aba7f"}
Sep 11 16:35:29.513613 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.513593    2577 scope.go:117] "RemoveContainer" containerID="0a680931a955b912451051d8b55a5fbf25f4203294cdfdd8a5fde767d01aba7f"
Sep 11 16:35:29.515110 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.515091    2577 generic.go:356] "Generic (PLEG): container finished" podID="415703dd-3c9d-487b-a384-2dee750c0915" containerID="4d6f5fbb5133b03bc35263afb37c02827cbed6cffa87711e11e4b856c268ccab" exitCode=1
Sep 11 16:35:29.515206 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.515162    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77" event={"ID":"415703dd-3c9d-487b-a384-2dee750c0915","Type":"ContainerDied","Data":"4d6f5fbb5133b03bc35263afb37c02827cbed6cffa87711e11e4b856c268ccab"}
Sep 11 16:35:29.515466 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.515448    2577 scope.go:117] "RemoveContainer" containerID="4d6f5fbb5133b03bc35263afb37c02827cbed6cffa87711e11e4b856c268ccab"
Sep 11 16:35:29.516872 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.516854    2577 generic.go:356] "Generic (PLEG): container finished" podID="3e4cb213-1072-4f8d-9199-9f4c0476c454" containerID="8e36de8a34c8b50cf29ea9266b7b6d8ba43ed514aaa828504e05604226eed7e1" exitCode=1
Sep 11 16:35:29.516954 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.516885    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp" event={"ID":"3e4cb213-1072-4f8d-9199-9f4c0476c454","Type":"ContainerDied","Data":"8e36de8a34c8b50cf29ea9266b7b6d8ba43ed514aaa828504e05604226eed7e1"}
Sep 11 16:35:29.517183 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.517169    2577 scope.go:117] "RemoveContainer" containerID="8e36de8a34c8b50cf29ea9266b7b6d8ba43ed514aaa828504e05604226eed7e1"
Sep 11 16:35:29.519029 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.519011    2577 generic.go:356] "Generic (PLEG): container finished" podID="b403831a-b82b-486e-9c63-2b7382bb238e" containerID="1313368003d0e8ca824db09c7d88813fce957ac7654cb84966af3dbe4aceff53" exitCode=1
Sep 11 16:35:29.519029 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.519073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z" event={"ID":"b403831a-b82b-486e-9c63-2b7382bb238e","Type":"ContainerDied","Data":"1313368003d0e8ca824db09c7d88813fce957ac7654cb84966af3dbe4aceff53"}
Sep 11 16:35:29.519395 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.519379    2577 scope.go:117] "RemoveContainer" containerID="1313368003d0e8ca824db09c7d88813fce957ac7654cb84966af3dbe4aceff53"
Sep 11 16:35:29.521408 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.521387    2577 generic.go:356] "Generic (PLEG): container finished" podID="31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e" containerID="71647e42bbdde284a167e127caf5ada0c031bdf861571b2ac66782dd57051e9c" exitCode=1
Sep 11 16:35:29.521504 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.521491    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf" event={"ID":"31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e","Type":"ContainerDied","Data":"71647e42bbdde284a167e127caf5ada0c031bdf861571b2ac66782dd57051e9c"}
Sep 11 16:35:29.521783 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:29.521768    2577 scope.go:117] "RemoveContainer" containerID="71647e42bbdde284a167e127caf5ada0c031bdf861571b2ac66782dd57051e9c"
Sep 11 16:35:30.510518 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:30.510480    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm"
Sep 11 16:35:30.525470 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:35:30.525438    2577 kuberuntime_manager.go:1459] "Unhandled Error" err="container manager start failed in pod dns-operator-controller-manager-7b84cd7c9c-7p78z_kuadrant-system(b403831a-b82b-486e-9c63-2b7382bb238e): CreateContainerConfigError: failed to sync configmap cache: timed out waiting for the condition" logger="UnhandledError"
Sep 11 16:35:30.525610 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:35:30.525478    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CreateContainerConfigError: \"failed to sync configmap cache: timed out waiting for the condition\"" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z" podUID="b403831a-b82b-486e-9c63-2b7382bb238e"
Sep 11 16:35:30.529008 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:30.528980    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp" event={"ID":"3e4cb213-1072-4f8d-9199-9f4c0476c454","Type":"ContainerStarted","Data":"d384879ef9313e41c777ac5ef39022970d7424c3c365ee1533943a74c74a171d"}
Sep 11 16:35:30.529212 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:30.529198    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp"
Sep 11 16:35:30.530914 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:30.530893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf" event={"ID":"31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e","Type":"ContainerStarted","Data":"73519e96cd1f0a12663789143bf7027df40fed802313ac54ecc4c911319c284e"}
Sep 11 16:35:30.531094 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:30.531080    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf"
Sep 11 16:35:30.532907 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:30.532890    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c" event={"ID":"89912015-3e6c-41e8-8ce3-1af04df3584d","Type":"ContainerStarted","Data":"bde02b9fe8f7e73fe50ff560ff045fc0edef5447a4260ae575d23227abb07797"}
Sep 11 16:35:30.533086 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:30.533073    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c"
Sep 11 16:35:30.534721 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:30.534701    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-vncpm" event={"ID":"47a55b39-a458-4484-af8f-07a70f9de757","Type":"ContainerStarted","Data":"c5b6d041116d2bbfe279fa86fd759964a5c0e6f92e3c96e72d15c2bde2a6f97a"}
Sep 11 16:35:30.534885 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:30.534868    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-vncpm"
Sep 11 16:35:30.536756 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:30.536736    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm" event={"ID":"96a83de2-03be-4483-a321-7f71b53a3a1b","Type":"ContainerStarted","Data":"1e09b5747414eeb09a5bed5d3e0fbe2cf85fa0a2e72b6fabdb464c1f789e4d1d"}
Sep 11 16:35:30.536850 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:30.536783    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm"
Sep 11 16:35:30.539090 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:30.539073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz" event={"ID":"b8929d82-a6e4-4e3c-a9e5-8158a14652bb","Type":"ContainerStarted","Data":"d105eaf439ce701272066c37fa75ca72d28cc33787a85d9b79f4c0747064abf4"}
Sep 11 16:35:30.539246 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:30.539234    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:35:31.546288 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:31.546247    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77" event={"ID":"415703dd-3c9d-487b-a384-2dee750c0915","Type":"ContainerStarted","Data":"e177af2e321755f1993f1e7fd4c007cd7d1c7e928a194180a2afed63e87b3acd"}
Sep 11 16:35:31.547163 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:31.547142    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77"
Sep 11 16:35:33.409766 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:33.409733    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 16:35:34.835755 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:34.835719    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z"
Sep 11 16:35:34.836179 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:34.836095    2577 scope.go:117] "RemoveContainer" containerID="1313368003d0e8ca824db09c7d88813fce957ac7654cb84966af3dbe4aceff53"
Sep 11 16:35:34.836340 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:35:34.836321    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=dns-operator-controller-manager-7b84cd7c9c-7p78z_kuadrant-system(b403831a-b82b-486e-9c63-2b7382bb238e)\"" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z" podUID="b403831a-b82b-486e-9c63-2b7382bb238e"
Sep 11 16:35:39.386221 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:39.386179    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z"
Sep 11 16:35:39.386750 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:39.386532    2577 scope.go:117] "RemoveContainer" containerID="1313368003d0e8ca824db09c7d88813fce957ac7654cb84966af3dbe4aceff53"
Sep 11 16:35:39.586937 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:39.586902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z" event={"ID":"b403831a-b82b-486e-9c63-2b7382bb238e","Type":"ContainerStarted","Data":"4e8c1fb90f628d73990b235d1ff6aa36f368f45d3659cd6732ee3a478b7853da"}
Sep 11 16:35:41.548963 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:41.548875    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c"
Sep 11 16:35:41.548963 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:41.548969    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf"
Sep 11 16:35:41.549428 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:41.549056    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-vncpm"
Sep 11 16:35:41.549535 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:41.549517    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm"
Sep 11 16:35:41.549597 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:41.549585    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-58649dcd54-t5vqz"
Sep 11 16:35:49.587472 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:49.587432    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z"
Sep 11 16:35:49.589064 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:35:49.589022    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z"
Sep 11 16:36:01.556924 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:01.556894    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp"
Sep 11 16:36:02.552806 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:02.552773    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77"
Sep 11 16:36:20.890176 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:20.890149    2577 scope.go:117] "RemoveContainer" containerID="4164c464234dc0c3465b4753c06b47ec6378f533fb08b787705e8d1d63eeae68"
Sep 11 16:36:23.804742 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.804711    2577 generic.go:356] "Generic (PLEG): container finished" podID="415703dd-3c9d-487b-a384-2dee750c0915" containerID="e177af2e321755f1993f1e7fd4c007cd7d1c7e928a194180a2afed63e87b3acd" exitCode=1
Sep 11 16:36:23.805183 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.804779    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77" event={"ID":"415703dd-3c9d-487b-a384-2dee750c0915","Type":"ContainerDied","Data":"e177af2e321755f1993f1e7fd4c007cd7d1c7e928a194180a2afed63e87b3acd"}
Sep 11 16:36:23.805183 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.804872    2577 scope.go:117] "RemoveContainer" containerID="4d6f5fbb5133b03bc35263afb37c02827cbed6cffa87711e11e4b856c268ccab"
Sep 11 16:36:23.805307 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.805282    2577 scope.go:117] "RemoveContainer" containerID="e177af2e321755f1993f1e7fd4c007cd7d1c7e928a194180a2afed63e87b3acd"
Sep 11 16:36:23.805513 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:23.805489    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-hcm77_opendatahub(415703dd-3c9d-487b-a384-2dee750c0915)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77" podUID="415703dd-3c9d-487b-a384-2dee750c0915"
Sep 11 16:36:23.806886 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.806867    2577 generic.go:356] "Generic (PLEG): container finished" podID="3e4cb213-1072-4f8d-9199-9f4c0476c454" containerID="d384879ef9313e41c777ac5ef39022970d7424c3c365ee1533943a74c74a171d" exitCode=1
Sep 11 16:36:23.806959 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.806930    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp" event={"ID":"3e4cb213-1072-4f8d-9199-9f4c0476c454","Type":"ContainerDied","Data":"d384879ef9313e41c777ac5ef39022970d7424c3c365ee1533943a74c74a171d"}
Sep 11 16:36:23.807280 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.807264    2577 scope.go:117] "RemoveContainer" containerID="d384879ef9313e41c777ac5ef39022970d7424c3c365ee1533943a74c74a171d"
Sep 11 16:36:23.807473 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:23.807454    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-2kmcp_opendatahub(3e4cb213-1072-4f8d-9199-9f4c0476c454)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp" podUID="3e4cb213-1072-4f8d-9199-9f4c0476c454"
Sep 11 16:36:23.808943 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.808924    2577 generic.go:356] "Generic (PLEG): container finished" podID="b403831a-b82b-486e-9c63-2b7382bb238e" containerID="4e8c1fb90f628d73990b235d1ff6aa36f368f45d3659cd6732ee3a478b7853da" exitCode=1
Sep 11 16:36:23.809028 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.808985    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z" event={"ID":"b403831a-b82b-486e-9c63-2b7382bb238e","Type":"ContainerDied","Data":"4e8c1fb90f628d73990b235d1ff6aa36f368f45d3659cd6732ee3a478b7853da"}
Sep 11 16:36:23.809310 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.809294    2577 scope.go:117] "RemoveContainer" containerID="4e8c1fb90f628d73990b235d1ff6aa36f368f45d3659cd6732ee3a478b7853da"
Sep 11 16:36:23.809463 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:23.809441    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=dns-operator-controller-manager-7b84cd7c9c-7p78z_kuadrant-system(b403831a-b82b-486e-9c63-2b7382bb238e)\"" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z" podUID="b403831a-b82b-486e-9c63-2b7382bb238e"
Sep 11 16:36:23.811160 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.811143    2577 generic.go:356] "Generic (PLEG): container finished" podID="31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e" containerID="73519e96cd1f0a12663789143bf7027df40fed802313ac54ecc4c911319c284e" exitCode=1
Sep 11 16:36:23.811242 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.811216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf" event={"ID":"31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e","Type":"ContainerDied","Data":"73519e96cd1f0a12663789143bf7027df40fed802313ac54ecc4c911319c284e"}
Sep 11 16:36:23.811480 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.811464    2577 scope.go:117] "RemoveContainer" containerID="73519e96cd1f0a12663789143bf7027df40fed802313ac54ecc4c911319c284e"
Sep 11 16:36:23.811642 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:23.811623    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=limitador-operator-controller-manager-74495c69f8-pqfnf_kuadrant-system(31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e)\"" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf" podUID="31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e"
Sep 11 16:36:23.813170 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.813151    2577 generic.go:356] "Generic (PLEG): container finished" podID="89912015-3e6c-41e8-8ce3-1af04df3584d" containerID="bde02b9fe8f7e73fe50ff560ff045fc0edef5447a4260ae575d23227abb07797" exitCode=1
Sep 11 16:36:23.813257 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.813218    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c" event={"ID":"89912015-3e6c-41e8-8ce3-1af04df3584d","Type":"ContainerDied","Data":"bde02b9fe8f7e73fe50ff560ff045fc0edef5447a4260ae575d23227abb07797"}
Sep 11 16:36:23.813487 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.813470    2577 scope.go:117] "RemoveContainer" containerID="bde02b9fe8f7e73fe50ff560ff045fc0edef5447a4260ae575d23227abb07797"
Sep 11 16:36:23.813631 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:23.813615    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=authorino-operator-76b478bd78-zgb8c_kuadrant-system(89912015-3e6c-41e8-8ce3-1af04df3584d)\"" pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c" podUID="89912015-3e6c-41e8-8ce3-1af04df3584d"
Sep 11 16:36:23.814950 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.814934    2577 generic.go:356] "Generic (PLEG): container finished" podID="47a55b39-a458-4484-af8f-07a70f9de757" containerID="c5b6d041116d2bbfe279fa86fd759964a5c0e6f92e3c96e72d15c2bde2a6f97a" exitCode=1
Sep 11 16:36:23.815056 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.814962    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-vncpm" event={"ID":"47a55b39-a458-4484-af8f-07a70f9de757","Type":"ContainerDied","Data":"c5b6d041116d2bbfe279fa86fd759964a5c0e6f92e3c96e72d15c2bde2a6f97a"}
Sep 11 16:36:23.815244 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.815231    2577 scope.go:117] "RemoveContainer" containerID="c5b6d041116d2bbfe279fa86fd759964a5c0e6f92e3c96e72d15c2bde2a6f97a"
Sep 11 16:36:23.815368 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:23.815353    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=odh-model-controller-7998c67dc4-vncpm_opendatahub(47a55b39-a458-4484-af8f-07a70f9de757)\"" pod="opendatahub/odh-model-controller-7998c67dc4-vncpm" podUID="47a55b39-a458-4484-af8f-07a70f9de757"
Sep 11 16:36:23.816892 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.816875    2577 generic.go:356] "Generic (PLEG): container finished" podID="96a83de2-03be-4483-a321-7f71b53a3a1b" containerID="1e09b5747414eeb09a5bed5d3e0fbe2cf85fa0a2e72b6fabdb464c1f789e4d1d" exitCode=0
Sep 11 16:36:23.816972 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.816909    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm" event={"ID":"96a83de2-03be-4483-a321-7f71b53a3a1b","Type":"ContainerDied","Data":"1e09b5747414eeb09a5bed5d3e0fbe2cf85fa0a2e72b6fabdb464c1f789e4d1d"}
Sep 11 16:36:23.817148 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.817135    2577 scope.go:117] "RemoveContainer" containerID="1e09b5747414eeb09a5bed5d3e0fbe2cf85fa0a2e72b6fabdb464c1f789e4d1d"
Sep 11 16:36:23.817289 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:23.817274    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kuadrant-operator-controller-manager-5769f99d9f-6npsm_kuadrant-system(96a83de2-03be-4483-a321-7f71b53a3a1b)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm" podUID="96a83de2-03be-4483-a321-7f71b53a3a1b"
Sep 11 16:36:23.841835 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.841811    2577 scope.go:117] "RemoveContainer" containerID="8e36de8a34c8b50cf29ea9266b7b6d8ba43ed514aaa828504e05604226eed7e1"
Sep 11 16:36:23.853866 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.853846    2577 scope.go:117] "RemoveContainer" containerID="1313368003d0e8ca824db09c7d88813fce957ac7654cb84966af3dbe4aceff53"
Sep 11 16:36:23.865555 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.865533    2577 scope.go:117] "RemoveContainer" containerID="71647e42bbdde284a167e127caf5ada0c031bdf861571b2ac66782dd57051e9c"
Sep 11 16:36:23.877169 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.877142    2577 scope.go:117] "RemoveContainer" containerID="d247bb25b8df8d65d536890247acef3bf4add5dda37127dd333ba2dcd1547a02"
Sep 11 16:36:23.888755 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.888737    2577 scope.go:117] "RemoveContainer" containerID="4694e8a00e39848e9b8e750976de1c0234980eed356773926d574216a4ff4ae4"
Sep 11 16:36:23.900078 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:23.900051    2577 scope.go:117] "RemoveContainer" containerID="8ed23ec73f42eefb2cd9d28e11bf9dc4f8a5f6a635664b267b7e117a8be0032a"
Sep 11 16:36:26.547061 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:26.546998    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp"
Sep 11 16:36:26.547540 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:26.547348    2577 scope.go:117] "RemoveContainer" containerID="d384879ef9313e41c777ac5ef39022970d7424c3c365ee1533943a74c74a171d"
Sep 11 16:36:26.547540 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:26.547507    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-2kmcp_opendatahub(3e4cb213-1072-4f8d-9199-9f4c0476c454)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp" podUID="3e4cb213-1072-4f8d-9199-9f4c0476c454"
Sep 11 16:36:27.551601 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:27.551567    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77"
Sep 11 16:36:27.551990 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:27.551913    2577 scope.go:117] "RemoveContainer" containerID="e177af2e321755f1993f1e7fd4c007cd7d1c7e928a194180a2afed63e87b3acd"
Sep 11 16:36:27.552146 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:27.552126    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-hcm77_opendatahub(415703dd-3c9d-487b-a384-2dee750c0915)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77" podUID="415703dd-3c9d-487b-a384-2dee750c0915"
Sep 11 16:36:27.990207 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:27.990160    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/odh-model-controller-7998c67dc4-vncpm"
Sep 11 16:36:27.990518 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:27.990506    2577 scope.go:117] "RemoveContainer" containerID="c5b6d041116d2bbfe279fa86fd759964a5c0e6f92e3c96e72d15c2bde2a6f97a"
Sep 11 16:36:27.990678 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:27.990651    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=odh-model-controller-7998c67dc4-vncpm_opendatahub(47a55b39-a458-4484-af8f-07a70f9de757)\"" pod="opendatahub/odh-model-controller-7998c67dc4-vncpm" podUID="47a55b39-a458-4484-af8f-07a70f9de757"
Sep 11 16:36:28.675202 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:28.675170    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp"
Sep 11 16:36:28.675687 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:28.675587    2577 scope.go:117] "RemoveContainer" containerID="d384879ef9313e41c777ac5ef39022970d7424c3c365ee1533943a74c74a171d"
Sep 11 16:36:28.675825 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:28.675804    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-2kmcp_opendatahub(3e4cb213-1072-4f8d-9199-9f4c0476c454)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp" podUID="3e4cb213-1072-4f8d-9199-9f4c0476c454"
Sep 11 16:36:28.681681 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:28.681659    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77"
Sep 11 16:36:28.681991 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:28.681979    2577 scope.go:117] "RemoveContainer" containerID="e177af2e321755f1993f1e7fd4c007cd7d1c7e928a194180a2afed63e87b3acd"
Sep 11 16:36:28.682176 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:28.682159    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-hcm77_opendatahub(415703dd-3c9d-487b-a384-2dee750c0915)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77" podUID="415703dd-3c9d-487b-a384-2dee750c0915"
Sep 11 16:36:29.393431 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:29.393363    2577 controller.go:195] "Failed to update lease" err="Put \"https://a6166511cd9264054a0e0222758160b6-fe0a73c1a588f6e1.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-139-140.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 11 16:36:29.589643 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:29.589589    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z"
Sep 11 16:36:29.589941 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:29.589926    2577 scope.go:117] "RemoveContainer" containerID="4e8c1fb90f628d73990b235d1ff6aa36f368f45d3659cd6732ee3a478b7853da"
Sep 11 16:36:29.590162 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:29.590143    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=dns-operator-controller-manager-7b84cd7c9c-7p78z_kuadrant-system(b403831a-b82b-486e-9c63-2b7382bb238e)\"" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z" podUID="b403831a-b82b-486e-9c63-2b7382bb238e"
Sep 11 16:36:31.546607 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:31.546553    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c"
Sep 11 16:36:31.547102 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:31.546848    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf"
Sep 11 16:36:31.547102 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:31.546895    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm"
Sep 11 16:36:31.547102 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:31.546940    2577 scope.go:117] "RemoveContainer" containerID="bde02b9fe8f7e73fe50ff560ff045fc0edef5447a4260ae575d23227abb07797"
Sep 11 16:36:31.547275 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:31.547114    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-vncpm"
Sep 11 16:36:31.547275 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:31.547201    2577 scope.go:117] "RemoveContainer" containerID="1e09b5747414eeb09a5bed5d3e0fbe2cf85fa0a2e72b6fabdb464c1f789e4d1d"
Sep 11 16:36:31.547275 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:31.547201    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=authorino-operator-76b478bd78-zgb8c_kuadrant-system(89912015-3e6c-41e8-8ce3-1af04df3584d)\"" pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c" podUID="89912015-3e6c-41e8-8ce3-1af04df3584d"
Sep 11 16:36:31.547435 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:31.547344    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kuadrant-operator-controller-manager-5769f99d9f-6npsm_kuadrant-system(96a83de2-03be-4483-a321-7f71b53a3a1b)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm" podUID="96a83de2-03be-4483-a321-7f71b53a3a1b"
Sep 11 16:36:31.547435 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:31.547412    2577 scope.go:117] "RemoveContainer" containerID="73519e96cd1f0a12663789143bf7027df40fed802313ac54ecc4c911319c284e"
Sep 11 16:36:31.547544 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:31.547501    2577 scope.go:117] "RemoveContainer" containerID="c5b6d041116d2bbfe279fa86fd759964a5c0e6f92e3c96e72d15c2bde2a6f97a"
Sep 11 16:36:31.547622 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:31.547601    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=limitador-operator-controller-manager-74495c69f8-pqfnf_kuadrant-system(31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e)\"" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf" podUID="31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e"
Sep 11 16:36:31.547622 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:31.547612    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=odh-model-controller-7998c67dc4-vncpm_opendatahub(47a55b39-a458-4484-af8f-07a70f9de757)\"" pod="opendatahub/odh-model-controller-7998c67dc4-vncpm" podUID="47a55b39-a458-4484-af8f-07a70f9de757"
Sep 11 16:36:34.835314 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:34.835285    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z"
Sep 11 16:36:34.835684 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:34.835656    2577 scope.go:117] "RemoveContainer" containerID="4e8c1fb90f628d73990b235d1ff6aa36f368f45d3659cd6732ee3a478b7853da"
Sep 11 16:36:34.835881 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:34.835861    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=dns-operator-controller-manager-7b84cd7c9c-7p78z_kuadrant-system(b403831a-b82b-486e-9c63-2b7382bb238e)\"" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z" podUID="b403831a-b82b-486e-9c63-2b7382bb238e"
Sep 11 16:36:37.052502 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:37.052466    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf"
Sep 11 16:36:37.052898 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:37.052818    2577 scope.go:117] "RemoveContainer" containerID="73519e96cd1f0a12663789143bf7027df40fed802313ac54ecc4c911319c284e"
Sep 11 16:36:37.895959 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:37.895924    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf" event={"ID":"31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e","Type":"ContainerStarted","Data":"c3e725b5b54d7b9bcc420d83e0476cb97bf561d2249dc417b0791a017800856c"}
Sep 11 16:36:37.896265 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:37.896149    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf"
Sep 11 16:36:38.764687 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:38.764645    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm"
Sep 11 16:36:38.765106 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:38.765072    2577 scope.go:117] "RemoveContainer" containerID="1e09b5747414eeb09a5bed5d3e0fbe2cf85fa0a2e72b6fabdb464c1f789e4d1d"
Sep 11 16:36:38.902894 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:38.902866    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm" event={"ID":"96a83de2-03be-4483-a321-7f71b53a3a1b","Type":"ContainerStarted","Data":"625d1009ebbdd0ead2af7b8608cad60fdcf91816202815e5744a3ff567fb5f21"}
Sep 11 16:36:38.903180 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:38.903162    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm"
Sep 11 16:36:39.394142 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:39.394089    2577 controller.go:195] "Failed to update lease" err="Put \"https://a6166511cd9264054a0e0222758160b6-fe0a73c1a588f6e1.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-139-140.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 11 16:36:40.626999 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:40.626962    2577 scope.go:117] "RemoveContainer" containerID="e177af2e321755f1993f1e7fd4c007cd7d1c7e928a194180a2afed63e87b3acd"
Sep 11 16:36:41.627250 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:41.627170    2577 scope.go:117] "RemoveContainer" containerID="d384879ef9313e41c777ac5ef39022970d7424c3c365ee1533943a74c74a171d"
Sep 11 16:36:41.919624 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:41.919542    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77" event={"ID":"415703dd-3c9d-487b-a384-2dee750c0915","Type":"ContainerStarted","Data":"7633948e25dc7f32488bbda5903b5eaebdae1a222842c8ce435a5cb902b2fce8"}
Sep 11 16:36:41.919770 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:41.919749    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77"
Sep 11 16:36:42.264747 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:42.264723    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c"
Sep 11 16:36:42.265136 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:42.265120    2577 scope.go:117] "RemoveContainer" containerID="bde02b9fe8f7e73fe50ff560ff045fc0edef5447a4260ae575d23227abb07797"
Sep 11 16:36:42.926544 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:42.926510    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp" event={"ID":"3e4cb213-1072-4f8d-9199-9f4c0476c454","Type":"ContainerStarted","Data":"9446670f26e27be0c91a95c2d56e8ee33ad354c9b17b7f162cd3f0dc0949ea84"}
Sep 11 16:36:42.926997 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:42.926714    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp"
Sep 11 16:36:42.928481 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:42.928459    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c" event={"ID":"89912015-3e6c-41e8-8ce3-1af04df3584d","Type":"ContainerStarted","Data":"0418f85da2d6c3f546c176b77fa56285a5913c1657bd7c1986dc9c601dc66ce2"}
Sep 11 16:36:42.928722 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:42.928705    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c"
Sep 11 16:36:43.626472 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:43.626444    2577 scope.go:117] "RemoveContainer" containerID="c5b6d041116d2bbfe279fa86fd759964a5c0e6f92e3c96e72d15c2bde2a6f97a"
Sep 11 16:36:44.939339 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:44.939302    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-vncpm" event={"ID":"47a55b39-a458-4484-af8f-07a70f9de757","Type":"ContainerStarted","Data":"6736de053b116fca1669851ee9d2d5bd425a4c9a68eb32791aee135c5fb3eb2c"}
Sep 11 16:36:44.939811 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:44.939535    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-vncpm"
Sep 11 16:36:46.626980 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:46.626944    2577 scope.go:117] "RemoveContainer" containerID="4e8c1fb90f628d73990b235d1ff6aa36f368f45d3659cd6732ee3a478b7853da"
Sep 11 16:36:46.950223 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:46.950130    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z" event={"ID":"b403831a-b82b-486e-9c63-2b7382bb238e","Type":"ContainerStarted","Data":"cd874f98b84bb735deba7fd910ab1f018978b211f90d717a77a3bc3afe1cdf23"}
Sep 11 16:36:46.950396 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:46.950365    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z"
Sep 11 16:36:48.905178 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:48.905146    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-pqfnf"
Sep 11 16:36:49.395341 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:49.395296    2577 controller.go:195] "Failed to update lease" err="Put \"https://a6166511cd9264054a0e0222758160b6-fe0a73c1a588f6e1.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-139-140.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 11 16:36:49.911174 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:49.911143    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6npsm"
Sep 11 16:36:51.883160 ip-10-0-139-140 kubenswrapper[2577]: E0911 16:36:51.883126    2577 controller.go:195] "Failed to update lease" err="Operation cannot be fulfilled on leases.coordination.k8s.io \"ip-10-0-139-140.ec2.internal\": the object has been modified; please apply your changes to the latest version and try again"
Sep 11 16:36:53.935196 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:53.935160    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-zgb8c"
Sep 11 16:36:55.946837 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:55.946803    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-vncpm"
Sep 11 16:36:57.958361 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:36:57.958328    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7p78z"
Sep 11 16:37:12.931064 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:12.931017    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcm77"
Sep 11 16:37:13.938868 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:13.938836    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-2kmcp"
Sep 11 16:37:16.044146 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:16.044106    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55fd49fbf5-hctcb_e2256359-778b-44b4-8956-62a5a4626466/authorino/0.log"
Sep 11 16:37:20.700659 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:20.700628    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/1.log"
Sep 11 16:37:20.704422 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:20.704398    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/1.log"
Sep 11 16:37:20.707374 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:20.707351    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ng2ww_07d57c04-0697-419b-9d3e-ca11761da85c/ovn-acl-logging/0.log"
Sep 11 16:37:20.711506 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:20.711485    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ng2ww_07d57c04-0697-419b-9d3e-ca11761da85c/ovn-acl-logging/0.log"
Sep 11 16:37:22.142416 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:22.142385    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-2kmcp_3e4cb213-1072-4f8d-9199-9f4c0476c454/manager/2.log"
Sep 11 16:37:22.257889 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:22.257855    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hcm77_415703dd-3c9d-487b-a384-2dee750c0915/manager/2.log"
Sep 11 16:37:22.585424 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:22.585354    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-nzzfd_65b17b5a-de65-4253-8a1b-de2adb5ae416/server/0.log"
Sep 11 16:37:22.715636 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:22.715608    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-vncpm_47a55b39-a458-4484-af8f-07a70f9de757/manager/2.log"
Sep 11 16:37:23.150010 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:23.149983    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58649dcd54-t5vqz_b8929d82-a6e4-4e3c-a9e5-8158a14652bb/copy-manifests/0.log"
Sep 11 16:37:23.166220 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:23.166191    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58649dcd54-t5vqz_b8929d82-a6e4-4e3c-a9e5-8158a14652bb/manager/1.log"
Sep 11 16:37:23.564315 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:23.564233    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-845964844c-8cfwp_c138bc01-6529-4072-813f-d5102dca4d1b/maas-api/0.log"
Sep 11 16:37:24.010914 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:24.010883    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-ll8j7_cdfd0296-4092-4b0c-b2db-6c748bcd32ad/postgres/0.log"
Sep 11 16:37:24.805679 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:24.805646    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55fd49fbf5-hctcb_e2256359-778b-44b4-8956-62a5a4626466/authorino/0.log"
Sep 11 16:37:24.963960 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:24.963932    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-zgb8c_89912015-3e6c-41e8-8ce3-1af04df3584d/manager/2.log"
Sep 11 16:37:25.094569 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:25.094488    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-7p78z_b403831a-b82b-486e-9c63-2b7382bb238e/manager/2.log"
Sep 11 16:37:25.340085 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:25.340057    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-6npsm_96a83de2-03be-4483-a321-7f71b53a3a1b/manager/2.log"
Sep 11 16:37:25.684555 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:25.684520    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-pqfnf_31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e/manager/2.log"
Sep 11 16:37:26.474081 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:26.474022    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5864499db4-xmxgc_0bf2e4aa-7611-4ae4-b9d2-16c18d838d14/kube-auth-proxy/0.log"
Sep 11 16:37:26.767834 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:26.767748    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fl68q_77c07ed6-da3e-4ab4-a7d7-ba7de106c633/payload-pre-processing/0.log"
Sep 11 16:37:27.698019 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:27.697988    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-lb9kc_fb15cdc1-cc17-4c1b-bec5-b757c56bcef8/main/0.log"
Sep 11 16:37:27.824660 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:27.824627    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-4nzq2_a8bac3d7-6c0a-4e54-a454-fc78bf587340/main/0.log"
Sep 11 16:37:28.097781 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:28.097758    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-jfp6b_9b97fcf2-48f1-4eb7-9bc1-7b71fb23f61b/main/0.log"
Sep 11 16:37:39.044528 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:39.044501    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-s8699_14f2dae2-6830-49c3-8f40-786b7f33ec2d/global-pull-secret-syncer/0.log"
Sep 11 16:37:39.511815 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:39.511784    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d2xpr_4c9f7e6e-73fc-4d05-b2a3-bffad79fa6cf/konnectivity-agent/0.log"
Sep 11 16:37:40.032054 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:40.032005    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-140.ec2.internal_615418bae492da132dc002c225bf9501/haproxy/0.log"
Sep 11 16:37:43.133184 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:43.133153    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55fd49fbf5-hctcb_e2256359-778b-44b4-8956-62a5a4626466/authorino/0.log"
Sep 11 16:37:43.281529 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:43.281500    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-zgb8c_89912015-3e6c-41e8-8ce3-1af04df3584d/manager/1.log"
Sep 11 16:37:43.281529 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:43.281534    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-zgb8c_89912015-3e6c-41e8-8ce3-1af04df3584d/manager/2.log"
Sep 11 16:37:43.346887 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:43.346853    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-7p78z_b403831a-b82b-486e-9c63-2b7382bb238e/manager/2.log"
Sep 11 16:37:43.347092 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:43.346940    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-7p78z_b403831a-b82b-486e-9c63-2b7382bb238e/manager/1.log"
Sep 11 16:37:43.474512 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:43.474434    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-6npsm_96a83de2-03be-4483-a321-7f71b53a3a1b/manager/1.log"
Sep 11 16:37:43.475757 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:43.475738    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-6npsm_96a83de2-03be-4483-a321-7f71b53a3a1b/manager/2.log"
Sep 11 16:37:43.603084 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:43.603023    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-pqfnf_31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e/manager/1.log"
Sep 11 16:37:43.603240 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:43.603227    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-pqfnf_31e7d2d8-7ec6-4ef0-8c87-fb40b09f761e/manager/2.log"
Sep 11 16:37:46.690748 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:46.690713    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l8dfk_57c804d5-daad-45c9-96c0-488161d776da/node-exporter/0.log"
Sep 11 16:37:46.714585 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:46.714562    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l8dfk_57c804d5-daad-45c9-96c0-488161d776da/kube-rbac-proxy/0.log"
Sep 11 16:37:46.775701 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:46.775676    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l8dfk_57c804d5-daad-45c9-96c0-488161d776da/init-textfile/0.log"
Sep 11 16:37:48.063692 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.063660    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"]
Sep 11 16:37:48.064110 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.063990    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20cce86b-af5b-46ba-a3dd-f38f288f83fe" containerName="payload-processing"
Sep 11 16:37:48.064110 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.064000    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="20cce86b-af5b-46ba-a3dd-f38f288f83fe" containerName="payload-processing"
Sep 11 16:37:48.064110 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.064009    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8" containerName="maas-api"
Sep 11 16:37:48.064110 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.064015    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8" containerName="maas-api"
Sep 11 16:37:48.064110 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.064029    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6897d110-2a89-4930-92f3-c74204aff896" containerName="payload-pre-processing"
Sep 11 16:37:48.064110 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.064051    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6897d110-2a89-4930-92f3-c74204aff896" containerName="payload-pre-processing"
Sep 11 16:37:48.064110 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.064064    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90b041b5-ada9-442c-9e74-d865a4bff30a" containerName="payload-pre-processing"
Sep 11 16:37:48.064110 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.064070    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="90b041b5-ada9-442c-9e74-d865a4bff30a" containerName="payload-pre-processing"
Sep 11 16:37:48.064110 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.064082    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63051b3a-b943-4742-9ee1-a48d2ae7f28a" containerName="payload-processing"
Sep 11 16:37:48.064110 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.064087    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="63051b3a-b943-4742-9ee1-a48d2ae7f28a" containerName="payload-processing"
Sep 11 16:37:48.064110 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.064099    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4311d371-c6e8-4d66-8bce-fbea05486610" containerName="payload-pre-processing"
Sep 11 16:37:48.064110 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.064104    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4311d371-c6e8-4d66-8bce-fbea05486610" containerName="payload-pre-processing"
Sep 11 16:37:48.064110 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.064111    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0df127-ec55-4412-8e80-579ffb7a0cc6" containerName="payload-processing"
Sep 11 16:37:48.064110 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.064117    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0df127-ec55-4412-8e80-579ffb7a0cc6" containerName="payload-processing"
Sep 11 16:37:48.064569 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.064169    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="90b041b5-ada9-442c-9e74-d865a4bff30a" containerName="payload-pre-processing"
Sep 11 16:37:48.064569 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.064179    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b0df127-ec55-4412-8e80-579ffb7a0cc6" containerName="payload-processing"
Sep 11 16:37:48.064569 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.064185    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6897d110-2a89-4930-92f3-c74204aff896" containerName="payload-pre-processing"
Sep 11 16:37:48.064569 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.064192    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d34c6fe2-c2a0-4854-98a2-cbc699d3cdd8" containerName="maas-api"
Sep 11 16:37:48.064569 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.064196    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="20cce86b-af5b-46ba-a3dd-f38f288f83fe" containerName="payload-processing"
Sep 11 16:37:48.064569 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.064203    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4311d371-c6e8-4d66-8bce-fbea05486610" containerName="payload-pre-processing"
Sep 11 16:37:48.064569 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.064207    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="63051b3a-b943-4742-9ee1-a48d2ae7f28a" containerName="payload-processing"
Sep 11 16:37:48.067298 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.067281    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"
Sep 11 16:37:48.071791 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.071768    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jh25b\"/\"default-dockercfg-t8w88\""
Sep 11 16:37:48.072720 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.072703    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jh25b\"/\"kube-root-ca.crt\""
Sep 11 16:37:48.072814 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.072762    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jh25b\"/\"openshift-service-ca.crt\""
Sep 11 16:37:48.086599 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.086578    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"]
Sep 11 16:37:48.168393 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.168347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37d97cce-2cd0-45c9-83ee-03ac88637bfc-podres\") pod \"perf-node-gather-daemonset-j2598\" (UID: \"37d97cce-2cd0-45c9-83ee-03ac88637bfc\") " pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"
Sep 11 16:37:48.168567 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.168450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37d97cce-2cd0-45c9-83ee-03ac88637bfc-sys\") pod \"perf-node-gather-daemonset-j2598\" (UID: \"37d97cce-2cd0-45c9-83ee-03ac88637bfc\") " pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"
Sep 11 16:37:48.168567 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.168501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37d97cce-2cd0-45c9-83ee-03ac88637bfc-lib-modules\") pod \"perf-node-gather-daemonset-j2598\" (UID: \"37d97cce-2cd0-45c9-83ee-03ac88637bfc\") " pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"
Sep 11 16:37:48.168567 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.168539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbpqn\" (UniqueName: \"kubernetes.io/projected/37d97cce-2cd0-45c9-83ee-03ac88637bfc-kube-api-access-cbpqn\") pod \"perf-node-gather-daemonset-j2598\" (UID: \"37d97cce-2cd0-45c9-83ee-03ac88637bfc\") " pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"
Sep 11 16:37:48.168672 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.168597    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37d97cce-2cd0-45c9-83ee-03ac88637bfc-proc\") pod \"perf-node-gather-daemonset-j2598\" (UID: \"37d97cce-2cd0-45c9-83ee-03ac88637bfc\") " pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"
Sep 11 16:37:48.269818 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.269784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37d97cce-2cd0-45c9-83ee-03ac88637bfc-lib-modules\") pod \"perf-node-gather-daemonset-j2598\" (UID: \"37d97cce-2cd0-45c9-83ee-03ac88637bfc\") " pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"
Sep 11 16:37:48.270021 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.269826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbpqn\" (UniqueName: \"kubernetes.io/projected/37d97cce-2cd0-45c9-83ee-03ac88637bfc-kube-api-access-cbpqn\") pod \"perf-node-gather-daemonset-j2598\" (UID: \"37d97cce-2cd0-45c9-83ee-03ac88637bfc\") " pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"
Sep 11 16:37:48.270021 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.269874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37d97cce-2cd0-45c9-83ee-03ac88637bfc-proc\") pod \"perf-node-gather-daemonset-j2598\" (UID: \"37d97cce-2cd0-45c9-83ee-03ac88637bfc\") " pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"
Sep 11 16:37:48.270021 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.269903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37d97cce-2cd0-45c9-83ee-03ac88637bfc-podres\") pod \"perf-node-gather-daemonset-j2598\" (UID: \"37d97cce-2cd0-45c9-83ee-03ac88637bfc\") " pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"
Sep 11 16:37:48.270021 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.270009    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37d97cce-2cd0-45c9-83ee-03ac88637bfc-lib-modules\") pod \"perf-node-gather-daemonset-j2598\" (UID: \"37d97cce-2cd0-45c9-83ee-03ac88637bfc\") " pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"
Sep 11 16:37:48.270273 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.270021    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37d97cce-2cd0-45c9-83ee-03ac88637bfc-proc\") pod \"perf-node-gather-daemonset-j2598\" (UID: \"37d97cce-2cd0-45c9-83ee-03ac88637bfc\") " pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"
Sep 11 16:37:48.270273 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.270064    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37d97cce-2cd0-45c9-83ee-03ac88637bfc-podres\") pod \"perf-node-gather-daemonset-j2598\" (UID: \"37d97cce-2cd0-45c9-83ee-03ac88637bfc\") " pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"
Sep 11 16:37:48.270273 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.270080    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37d97cce-2cd0-45c9-83ee-03ac88637bfc-sys\") pod \"perf-node-gather-daemonset-j2598\" (UID: \"37d97cce-2cd0-45c9-83ee-03ac88637bfc\") " pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"
Sep 11 16:37:48.270273 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.270155    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37d97cce-2cd0-45c9-83ee-03ac88637bfc-sys\") pod \"perf-node-gather-daemonset-j2598\" (UID: \"37d97cce-2cd0-45c9-83ee-03ac88637bfc\") " pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"
Sep 11 16:37:48.279930 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.279911    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbpqn\" (UniqueName: \"kubernetes.io/projected/37d97cce-2cd0-45c9-83ee-03ac88637bfc-kube-api-access-cbpqn\") pod \"perf-node-gather-daemonset-j2598\" (UID: \"37d97cce-2cd0-45c9-83ee-03ac88637bfc\") " pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"
Sep 11 16:37:48.377060 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.376999    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"
Sep 11 16:37:48.654785 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:48.654708    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"]
Sep 11 16:37:48.658226 ip-10-0-139-140 kubenswrapper[2577]: W0911 16:37:48.658200    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod37d97cce_2cd0_45c9_83ee_03ac88637bfc.slice/crio-82a99686aacfa131647d913420bb42adf5b67a67699774c13369f85c303b788d WatchSource:0}: Error finding container 82a99686aacfa131647d913420bb42adf5b67a67699774c13369f85c303b788d: Status 404 returned error can't find the container with id 82a99686aacfa131647d913420bb42adf5b67a67699774c13369f85c303b788d
Sep 11 16:37:49.254119 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:49.254076    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598" event={"ID":"37d97cce-2cd0-45c9-83ee-03ac88637bfc","Type":"ContainerStarted","Data":"bc5ad36dd439a30d437e20ae9d3c670c38752eab816cefeea83d1b1e45b7142d"}
Sep 11 16:37:49.254542 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:49.254135    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598" event={"ID":"37d97cce-2cd0-45c9-83ee-03ac88637bfc","Type":"ContainerStarted","Data":"82a99686aacfa131647d913420bb42adf5b67a67699774c13369f85c303b788d"}
Sep 11 16:37:49.254542 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:49.254175    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"
Sep 11 16:37:49.274273 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:49.274225    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598" podStartSLOduration=1.274209733 podStartE2EDuration="1.274209733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:37:49.272940562 +0000 UTC m=+2129.118584116" watchObservedRunningTime="2026-09-11 16:37:49.274209733 +0000 UTC m=+2129.119853287"
Sep 11 16:37:49.857025 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:49.856997    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/1.log"
Sep 11 16:37:49.862124 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:49.862094    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-v5mzm_286b445a-e066-44bb-8e69-cd3b0d0ccb30/console-operator/2.log"
Sep 11 16:37:51.143649 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:51.143617    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-fmmjp_16e8c31c-1064-4b5b-a1c9-1d2e509f6853/volume-data-source-validator/0.log"
Sep 11 16:37:52.403127 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:52.403097    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zqfcd_dafc1233-f30d-40de-9736-3cef5d49666f/dns/0.log"
Sep 11 16:37:52.428748 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:52.428720    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zqfcd_dafc1233-f30d-40de-9736-3cef5d49666f/kube-rbac-proxy/0.log"
Sep 11 16:37:52.456369 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:52.456328    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mnrzt_a790ffd9-de74-4a62-9e37-d0ab20455b53/dns-node-resolver/0.log"
Sep 11 16:37:53.485541 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:53.485516    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hlqbr_58ea2e2b-bb7b-4537-aebf-68daed411f86/node-ca/0.log"
Sep 11 16:37:55.251454 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:55.251426    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5864499db4-xmxgc_0bf2e4aa-7611-4ae4-b9d2-16c18d838d14/kube-auth-proxy/0.log"
Sep 11 16:37:55.269083 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:55.269057    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jh25b/perf-node-gather-daemonset-j2598"
Sep 11 16:37:55.397829 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:55.397799    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fl68q_77c07ed6-da3e-4ab4-a7d7-ba7de106c633/payload-pre-processing/0.log"
Sep 11 16:37:57.287512 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:57.287484    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-chv7q_2ba7b000-8761-42c8-9ef7-700cd3f1959f/serve-healthcheck-canary/0.log"
Sep 11 16:37:59.004649 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:59.004570    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c9g8w_07825a72-a9e3-47f9-a430-a64b648c5dbb/kube-rbac-proxy/0.log"
Sep 11 16:37:59.230995 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:59.230968    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c9g8w_07825a72-a9e3-47f9-a430-a64b648c5dbb/exporter/0.log"
Sep 11 16:37:59.308028 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:37:59.308001    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c9g8w_07825a72-a9e3-47f9-a430-a64b648c5dbb/extractor/0.log"
Sep 11 16:38:00.133752 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:38:00.133721    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5864499db4-xmxgc_0bf2e4aa-7611-4ae4-b9d2-16c18d838d14/kube-auth-proxy/0.log"
Sep 11 16:38:00.236351 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:38:00.236325    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fl68q_77c07ed6-da3e-4ab4-a7d7-ba7de106c633/payload-pre-processing/0.log"
Sep 11 16:38:03.086377 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:38:03.086348    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5864499db4-xmxgc_0bf2e4aa-7611-4ae4-b9d2-16c18d838d14/kube-auth-proxy/0.log"
Sep 11 16:38:03.220556 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:38:03.220527    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fl68q_77c07ed6-da3e-4ab4-a7d7-ba7de106c633/payload-pre-processing/0.log"
Sep 11 16:38:03.834323 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:38:03.834292    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-2kmcp_3e4cb213-1072-4f8d-9199-9f4c0476c454/manager/2.log"
Sep 11 16:38:03.834323 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:38:03.834310    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-2kmcp_3e4cb213-1072-4f8d-9199-9f4c0476c454/manager/1.log"
Sep 11 16:38:04.033028 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:38:04.033000    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hcm77_415703dd-3c9d-487b-a384-2dee750c0915/manager/2.log"
Sep 11 16:38:04.041165 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:38:04.041140    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hcm77_415703dd-3c9d-487b-a384-2dee750c0915/manager/1.log"
Sep 11 16:38:04.406543 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:38:04.406516    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-nzzfd_65b17b5a-de65-4253-8a1b-de2adb5ae416/server/0.log"
Sep 11 16:38:04.705363 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:38:04.705274    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-vncpm_47a55b39-a458-4484-af8f-07a70f9de757/manager/1.log"
Sep 11 16:38:04.705363 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:38:04.705316    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-vncpm_47a55b39-a458-4484-af8f-07a70f9de757/manager/2.log"
Sep 11 16:38:05.773777 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:38:05.773746    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58649dcd54-t5vqz_b8929d82-a6e4-4e3c-a9e5-8158a14652bb/manager/1.log"
Sep 11 16:38:05.814940 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:38:05.814908    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58649dcd54-t5vqz_b8929d82-a6e4-4e3c-a9e5-8158a14652bb/manager/0.log"
Sep 11 16:38:06.161412 ip-10-0-139-140 kubenswrapper[2577]: I0911 16:38:06.161385    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58649dcd54-t5vqz_b8929d82-a6e4-4e3c-a9e5-8158a14652bb/copy-manifests/0.log"