Sep 15 11:23:16.463134 ip-10-0-135-143 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 15 11:23:16.463149 ip-10-0-135-143 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 15 11:23:16.463159 ip-10-0-135-143 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 15 11:23:16.463474 ip-10-0-135-143 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 15 11:23:25.872830 ip-10-0-135-143 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 15 11:23:25.872846 ip-10-0-135-143 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 0843849261b64dd198d6ffa902dfd534 --
Sep 15 11:25:01.227353 ip-10-0-135-143 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 11:25:01.656239 ip-10-0-135-143 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 11:25:01.656239 ip-10-0-135-143 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 11:25:01.656239 ip-10-0-135-143 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 11:25:01.656239 ip-10-0-135-143 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 11:25:01.656239 ip-10-0-135-143 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 11:25:01.656957 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.656323    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 11:25:01.661923 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661901    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 11:25:01.661923 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661920    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 11:25:01.661923 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661925    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 11:25:01.661923 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661929    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 11:25:01.661923 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661932    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 11:25:01.662143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661935    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 11:25:01.662143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661939    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 11:25:01.662143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661942    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 11:25:01.662143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661945    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 11:25:01.662143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661947    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 11:25:01.662143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661950    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 11:25:01.662143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661953    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 11:25:01.662143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661955    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 11:25:01.662143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661958    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 11:25:01.662143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661961    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 11:25:01.662143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661963    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 11:25:01.662143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661966    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 11:25:01.662143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661969    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 11:25:01.662143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661971    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 11:25:01.662143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661974    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 11:25:01.662143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661980    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 11:25:01.662143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661982    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 11:25:01.662143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661985    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 11:25:01.662143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661988    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 11:25:01.662143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661990    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 11:25:01.662624 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661993    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 11:25:01.662624 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661995    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 11:25:01.662624 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.661998    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 11:25:01.662624 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662001    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 11:25:01.662624 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662004    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 11:25:01.662624 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662006    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 11:25:01.662624 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662009    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 11:25:01.662624 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662012    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 11:25:01.662624 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662015    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 11:25:01.662624 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662017    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 11:25:01.662624 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662020    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 11:25:01.662624 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662023    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 11:25:01.662624 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662025    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 11:25:01.662624 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662028    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 11:25:01.662624 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662031    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 11:25:01.662624 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662034    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 11:25:01.662624 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662037    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 11:25:01.662624 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662039    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 11:25:01.662624 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662042    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 11:25:01.662624 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662044    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 11:25:01.663129 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662047    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 11:25:01.663129 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662049    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 11:25:01.663129 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662052    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 11:25:01.663129 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662066    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 11:25:01.663129 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662069    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 11:25:01.663129 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662071    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 11:25:01.663129 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662074    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 11:25:01.663129 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662077    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 11:25:01.663129 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662080    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 11:25:01.663129 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662084    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 11:25:01.663129 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662087    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 11:25:01.663129 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662091    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 11:25:01.663129 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662093    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 11:25:01.663129 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662097    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 11:25:01.663129 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662100    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 11:25:01.663129 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662102    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 11:25:01.663129 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662105    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 11:25:01.663129 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662107    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 11:25:01.663129 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662110    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 11:25:01.663589 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662113    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 11:25:01.663589 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662116    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 11:25:01.663589 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662118    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 11:25:01.663589 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662121    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 11:25:01.663589 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662124    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 11:25:01.663589 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662129    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 11:25:01.663589 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662134    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 11:25:01.663589 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662137    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 11:25:01.663589 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662140    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 11:25:01.663589 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662143    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 11:25:01.663589 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662146    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 11:25:01.663589 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662149    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 11:25:01.663589 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662152    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 11:25:01.663589 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662155    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 11:25:01.663589 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662158    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 11:25:01.663589 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662161    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 11:25:01.663589 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662164    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 11:25:01.663589 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662166    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 11:25:01.663589 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662169    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 11:25:01.663589 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662172    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 11:25:01.664147 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662174    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 11:25:01.664147 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662177    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 11:25:01.664147 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662179    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 11:25:01.664147 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662182    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 11:25:01.664147 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662184    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 11:25:01.664147 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662187    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 11:25:01.664147 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662190    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 11:25:01.664147 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662193    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 11:25:01.664147 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662195    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 11:25:01.664147 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662198    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 11:25:01.664147 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662201    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 11:25:01.664147 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662203    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 11:25:01.664147 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662206    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 11:25:01.664147 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662208    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 11:25:01.664147 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662211    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 11:25:01.664147 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662896    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 11:25:01.664147 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662902    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 11:25:01.664147 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662905    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 11:25:01.664147 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662909    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 11:25:01.664626 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662912    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 11:25:01.664626 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662914    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 11:25:01.664626 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662917    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 11:25:01.664626 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662920    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 11:25:01.664626 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662923    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 11:25:01.664626 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662926    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 11:25:01.664626 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662929    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 11:25:01.664626 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662931    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 11:25:01.664626 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662934    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 11:25:01.664626 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662937    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 11:25:01.664626 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662940    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 11:25:01.664626 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662942    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 11:25:01.664626 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662945    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 11:25:01.664626 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662950    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 11:25:01.664626 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662954    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 11:25:01.664626 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662957    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 11:25:01.664626 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662960    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 11:25:01.664626 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662962    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 11:25:01.664626 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662965    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 11:25:01.664626 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662968    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 11:25:01.665167 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662971    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 11:25:01.665167 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662973    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 11:25:01.665167 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662976    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 11:25:01.665167 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662978    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 11:25:01.665167 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662981    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 11:25:01.665167 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662983    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 11:25:01.665167 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662986    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 11:25:01.665167 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662988    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 11:25:01.665167 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662991    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 11:25:01.665167 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.662995    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 11:25:01.665167 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663000    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 11:25:01.665167 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663004    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 11:25:01.665167 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663007    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 11:25:01.665167 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663010    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 11:25:01.665167 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663012    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 11:25:01.665167 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663016    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 11:25:01.665167 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663018    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 11:25:01.665167 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663021    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 11:25:01.665167 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663024    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 11:25:01.665653 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663026    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 11:25:01.665653 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663029    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 11:25:01.665653 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663032    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 11:25:01.665653 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663034    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 11:25:01.665653 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663037    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 11:25:01.665653 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663039    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 11:25:01.665653 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663042    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 11:25:01.665653 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663045    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 11:25:01.665653 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663047    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 11:25:01.665653 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663050    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 11:25:01.665653 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663070    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 11:25:01.665653 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663073    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 11:25:01.665653 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663075    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 11:25:01.665653 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663078    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 11:25:01.665653 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663081    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 11:25:01.665653 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663084    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 11:25:01.665653 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663086    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 11:25:01.665653 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663090    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 11:25:01.665653 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663093    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 11:25:01.666128 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663095    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 11:25:01.666128 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663098    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 11:25:01.666128 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663101    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 11:25:01.666128 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663103    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 11:25:01.666128 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663106    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 11:25:01.666128 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663109    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 11:25:01.666128 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663112    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 11:25:01.666128 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663115    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 11:25:01.666128 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663117    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 11:25:01.666128 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663120    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 11:25:01.666128 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663123    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 11:25:01.666128 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663126    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 11:25:01.666128 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663128    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 11:25:01.666128 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663131    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 11:25:01.666128 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663134    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 11:25:01.666128 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663137    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 11:25:01.666128 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663139    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 11:25:01.666128 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663142    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 11:25:01.666128 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663145    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 11:25:01.666128 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663147    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 11:25:01.666628 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663150    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 11:25:01.666628 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663154    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 11:25:01.666628 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663157    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 11:25:01.666628 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663159    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 11:25:01.666628 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663162    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 11:25:01.666628 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663165    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 11:25:01.666628 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663167    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 11:25:01.666628 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663170    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 11:25:01.666628 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663173    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 11:25:01.666628 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663175    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 11:25:01.666628 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663180    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 11:25:01.666628 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663184    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 11:25:01.666628 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663187    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 11:25:01.666628 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663190    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 11:25:01.666628 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663193    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 11:25:01.666628 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663196    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 11:25:01.666628 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663199    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 11:25:01.666628 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663576    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 11:25:01.666628 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663581    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 11:25:01.667113 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663584    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 11:25:01.667113 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663587    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 11:25:01.667113 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663590    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 11:25:01.667113 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663593    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 11:25:01.667113 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663596    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 11:25:01.667113 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663600    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 11:25:01.667113 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663602    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 11:25:01.667113 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663605    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 11:25:01.667113 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663608    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 11:25:01.667113 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663610    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 11:25:01.667113 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663613    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 11:25:01.667113 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663615    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 11:25:01.667113 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663618    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 11:25:01.667113 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663620    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 11:25:01.667113 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663623    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 11:25:01.667113 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663626    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 11:25:01.667113 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663629    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 11:25:01.667113 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663631    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 11:25:01.667113 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663634    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 11:25:01.667113 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663637    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 11:25:01.667613 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663639    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 11:25:01.667613 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663642    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 11:25:01.667613 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663644    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 11:25:01.667613 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663647    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 11:25:01.667613 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663650    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 11:25:01.667613 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663652    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 11:25:01.667613 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663655    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 11:25:01.667613 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663657    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 11:25:01.667613 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663660    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 11:25:01.667613 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663663    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 11:25:01.667613 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663665    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 11:25:01.667613 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663668    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 11:25:01.667613 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663670    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 11:25:01.667613 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663673    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 11:25:01.667613 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663675    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 11:25:01.667613 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663678    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 11:25:01.667613 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663680    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 11:25:01.667613 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663683    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 11:25:01.667613 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663685    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 11:25:01.667613 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663688    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 11:25:01.668106 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663690    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 11:25:01.668106 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663692    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 11:25:01.668106 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663695    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 11:25:01.668106 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663697    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 11:25:01.668106 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663700    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 11:25:01.668106 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663702    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 11:25:01.668106 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663705    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 11:25:01.668106 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663711    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 11:25:01.668106 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663714    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 11:25:01.668106 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663716    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 11:25:01.668106 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663719    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 11:25:01.668106 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663721    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 11:25:01.668106 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663724    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 11:25:01.668106 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663727    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 11:25:01.668106 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663729    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 11:25:01.668106 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663732    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 11:25:01.668106 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663734    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 11:25:01.668106 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663736    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 11:25:01.668106 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663739    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 11:25:01.668106 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663742    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 11:25:01.668726 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663744    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 11:25:01.668726 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663748    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 11:25:01.668726 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663751    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 11:25:01.668726 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663753    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 11:25:01.668726 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663756    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 11:25:01.668726 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663759    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 11:25:01.668726 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663761    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 11:25:01.668726 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663764    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 11:25:01.668726 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663766    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 11:25:01.668726 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663769    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 11:25:01.668726 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663771    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 11:25:01.668726 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663774    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 11:25:01.668726 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663776    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 11:25:01.668726 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663780    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 11:25:01.668726 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663783    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 11:25:01.668726 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663785    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 11:25:01.668726 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663788    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 11:25:01.668726 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663790    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 11:25:01.668726 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663793    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 11:25:01.669237 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663801    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 11:25:01.669237 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663805    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 11:25:01.669237 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663808    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 11:25:01.669237 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663811    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 11:25:01.669237 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663814    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 11:25:01.669237 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663816    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 11:25:01.669237 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663819    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 11:25:01.669237 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663823    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 11:25:01.669237 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663826    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 11:25:01.669237 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663829    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 11:25:01.669237 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663832    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 11:25:01.669237 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663834    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 11:25:01.669237 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663837    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 11:25:01.669237 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663839    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 11:25:01.669237 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663844    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 11:25:01.669237 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663846    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 11:25:01.669237 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663849    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 11:25:01.669237 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.663851    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 11:25:01.669237 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.663923    2562 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 11:25:01.669237 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.663961    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.663969    2562 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.663977    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.663982    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.663985    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.663989    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.663994    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.663997    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664000    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664004    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664007    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664010    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664013    2562 flags.go:64] FLAG: --cgroup-root=""
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664018    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664021    2562 flags.go:64] FLAG: --client-ca-file=""
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664025    2562 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664028    2562 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664032    2562 flags.go:64] FLAG: --cluster-domain=""
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664035    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664038    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664041    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664045    2562 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664048    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 11:25:01.669728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664051    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664068    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664072    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664075    2562 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664080    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664083    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664086    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664089    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664093    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664096    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664100    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664103    2562 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664106    2562 flags.go:64] FLAG: --enable-server="true"
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664109    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664114    2562 flags.go:64] FLAG: --event-burst="100"
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664117    2562 flags.go:64] FLAG: --event-qps="50"
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664120    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664123    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664126    2562 flags.go:64] FLAG: --eviction-hard=""
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664129    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664133    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664136    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664140    2562 flags.go:64] FLAG: --eviction-soft=""
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664144    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664146    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 11:25:01.670315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664150    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664154    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664157    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664160    2562 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664163    2562 flags.go:64] FLAG: --feature-gates=""
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664167    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664170    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664173    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664176    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664179    2562 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664182    2562 flags.go:64] FLAG: --help="false"
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664187    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-135-143.ec2.internal"
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664190    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664193    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664196    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664201    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664204    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664207    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664210    2562 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664213    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664216    2562 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664219    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664222    2562 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664225    2562 flags.go:64] FLAG: --kube-reserved=""
Sep 15 11:25:01.670959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664228    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664231    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664234    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664237    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664240    2562 flags.go:64] FLAG: --lock-file=""
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664243    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664246    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664249    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664254    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664258    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664261    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664263    2562 flags.go:64] FLAG: --logging-format="text"
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664266    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664270    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664272    2562 flags.go:64] FLAG: --manifest-url=""
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664275    2562 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664279    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664282    2562 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664286    2562 flags.go:64] FLAG: --max-pods="110"
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664291    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664294    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664296    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664299    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664302    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664305    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 11:25:01.671600 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664308    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664315    2562 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664318    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664321    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664324    2562 flags.go:64] FLAG: --pod-cidr=""
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664327    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664333    2562 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664335    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664339    2562 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664341    2562 flags.go:64] FLAG: --port="10250"
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664344    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664347    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01ffcb8aab3afaf0e"
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664351    2562 flags.go:64] FLAG: --qos-reserved=""
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664354    2562 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664357    2562 flags.go:64] FLAG: --register-node="true"
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664360    2562 flags.go:64] FLAG: --register-with-taints=""
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664364    2562 flags.go:64] FLAG: --registry-burst="10"
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664367    2562 flags.go:64] FLAG: --registry-qps="5"
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664370    2562 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664373    2562 flags.go:64] FLAG: --reserved-memory=""
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664376    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664379    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664382    2562 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664385    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664388    2562 flags.go:64] FLAG: --runonce="false"
Sep 15 11:25:01.672234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664391    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664396    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664399    2562 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664402    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664405    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664416    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664420    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664423    2562 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664426    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664429    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664432    2562 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664435    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664438    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664441    2562 flags.go:64] FLAG: --system-cgroups=""
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664444    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664449    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664452    2562 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664454    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664458    2562 flags.go:64] FLAG: --tls-min-version=""
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664461    2562 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664464    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664467    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664470    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664473    2562 flags.go:64] FLAG: --v="2"
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664477    2562 flags.go:64] FLAG: --version="false"
Sep 15 11:25:01.672901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664482    2562 flags.go:64] FLAG: --vmodule=""
Sep 15 11:25:01.673518 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664486    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 11:25:01.673518 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664489    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 11:25:01.673518 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664602    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 11:25:01.673518 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664605    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 11:25:01.673518 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664608    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 11:25:01.673518 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664611    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 11:25:01.673518 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664615    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 11:25:01.673518 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664618    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 11:25:01.673518 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664622    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 11:25:01.673518 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664624    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 11:25:01.673518 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664627    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 11:25:01.673518 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664630    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 11:25:01.673518 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664633    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 11:25:01.673518 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664635    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 11:25:01.673518 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664638    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 11:25:01.673518 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664647    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 11:25:01.673518 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664650    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 11:25:01.673518 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664653    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 11:25:01.673518 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664655    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 11:25:01.673518 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664658    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 11:25:01.674001 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664660    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 11:25:01.674001 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664663    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 11:25:01.674001 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664666    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 11:25:01.674001 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664668    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 11:25:01.674001 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664671    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 11:25:01.674001 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664674    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 11:25:01.674001 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664676    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 11:25:01.674001 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664679    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 11:25:01.674001 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664682    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 11:25:01.674001 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664684    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 11:25:01.674001 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664687    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 11:25:01.674001 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664690    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 11:25:01.674001 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664693    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 11:25:01.674001 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664696    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 11:25:01.674001 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664698    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 11:25:01.674001 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664701    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 11:25:01.674001 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664704    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 11:25:01.674001 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664706    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 11:25:01.674001 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664709    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 11:25:01.674001 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664713    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 11:25:01.674534 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664716    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 11:25:01.674534 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664718    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 11:25:01.674534 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664721    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 11:25:01.674534 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664724    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 11:25:01.674534 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664726    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 11:25:01.674534 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664729    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 11:25:01.674534 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664731    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 11:25:01.674534 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664734    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 11:25:01.674534 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664737    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 11:25:01.674534 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664740    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 11:25:01.674534 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664742    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 11:25:01.674534 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664746    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 11:25:01.674534 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664749    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 11:25:01.674534 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664753    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 11:25:01.674534 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664755    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 11:25:01.674534 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664758    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 11:25:01.674534 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664761    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 11:25:01.674534 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664763    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 11:25:01.674534 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664768    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 11:25:01.675008 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664770    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 11:25:01.675008 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664773    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 11:25:01.675008 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664776    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 11:25:01.675008 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664779    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 11:25:01.675008 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664781    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 11:25:01.675008 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664784    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 11:25:01.675008 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664787    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 11:25:01.675008 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664790    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 11:25:01.675008 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664812    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 11:25:01.675008 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664816    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 11:25:01.675008 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664819    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 11:25:01.675008 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664822    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 11:25:01.675008 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664826    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 11:25:01.675008 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664828    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 11:25:01.675008 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664831    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 11:25:01.675008 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664834    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 11:25:01.675008 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664836    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 11:25:01.675008 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664840    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 11:25:01.675008 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664844    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 11:25:01.675008 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664847    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 11:25:01.675507 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664850    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 11:25:01.675507 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664853    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 11:25:01.675507 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664855    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 11:25:01.675507 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664858    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 11:25:01.675507 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664861    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 11:25:01.675507 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664863    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 11:25:01.675507 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664866    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 11:25:01.675507 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664869    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 11:25:01.675507 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664872    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 11:25:01.675507 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664874    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 11:25:01.675507 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664877    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 11:25:01.675507 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664881    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 11:25:01.675507 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664884    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 11:25:01.675507 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664887    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 11:25:01.675507 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664889    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 11:25:01.675507 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664892    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 11:25:01.675507 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664895    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 11:25:01.675507 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664897    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 11:25:01.675507 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664900    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 11:25:01.675507 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664903    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 11:25:01.676034 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664906    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 11:25:01.676034 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.664910    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 11:25:01.676034 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.664915    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 11:25:01.676034 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.672755    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 11:25:01.676034 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.672772    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 11:25:01.676034 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.672785    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 11:25:01.676034 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.672791    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 11:25:01.676034 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672863    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 11:25:01.676034 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672870    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 11:25:01.676034 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672874    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 11:25:01.676034 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672877    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 11:25:01.676034 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672880    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 11:25:01.676034 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672883    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 11:25:01.676034 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672886    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 11:25:01.676434 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672889    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 11:25:01.676434 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672892    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 11:25:01.676434 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672895    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 11:25:01.676434 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672897    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 11:25:01.676434 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672902    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 11:25:01.676434 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672905    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 11:25:01.676434 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672908    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 11:25:01.676434 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672911    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 11:25:01.676434 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672914    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 11:25:01.676434 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672917    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 11:25:01.676434 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672919    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 11:25:01.676434 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672922    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 11:25:01.676434 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672925    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 11:25:01.676434 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672927    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 11:25:01.676434 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672930    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 11:25:01.676434 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672933    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 11:25:01.676434 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672936    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 11:25:01.676434 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672945    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 11:25:01.676434 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672948    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 11:25:01.676434 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672950    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 11:25:01.676921 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672953    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 11:25:01.676921 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672956    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 11:25:01.676921 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672960    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 11:25:01.676921 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672963    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 11:25:01.676921 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672966    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 11:25:01.676921 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672969    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 11:25:01.676921 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672972    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 11:25:01.676921 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672975    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 11:25:01.676921 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672977    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 11:25:01.676921 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672980    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 11:25:01.676921 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672982    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 11:25:01.676921 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672985    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 11:25:01.676921 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672988    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 11:25:01.676921 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672991    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 11:25:01.676921 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672993    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 11:25:01.676921 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672996    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 11:25:01.676921 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.672998    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 11:25:01.676921 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673002    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 11:25:01.676921 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673006    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 11:25:01.676921 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673009    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 11:25:01.677476 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673011    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 11:25:01.677476 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673014    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 11:25:01.677476 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673017    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 11:25:01.677476 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673019    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 11:25:01.677476 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673022    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 11:25:01.677476 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673024    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 11:25:01.677476 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673027    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 11:25:01.677476 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673030    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 11:25:01.677476 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673033    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 11:25:01.677476 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673035    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 11:25:01.677476 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673043    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 11:25:01.677476 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673046    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 11:25:01.677476 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673050    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 11:25:01.677476 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673068    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 11:25:01.677476 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673074    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 11:25:01.677476 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673079    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 11:25:01.677476 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673082    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 11:25:01.677476 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673085    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 11:25:01.677476 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673088    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 11:25:01.677945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673090    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 11:25:01.677945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673093    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 11:25:01.677945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673096    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 11:25:01.677945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673098    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 11:25:01.677945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673101    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 11:25:01.677945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673103    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 11:25:01.677945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673106    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 11:25:01.677945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673109    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 11:25:01.677945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673111    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 11:25:01.677945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673114    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 11:25:01.677945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673116    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 11:25:01.677945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673119    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 11:25:01.677945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673121    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 11:25:01.677945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673124    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 11:25:01.677945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673127    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 11:25:01.677945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673129    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 11:25:01.677945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673132    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 11:25:01.677945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673134    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 11:25:01.677945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673137    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 11:25:01.677945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673139    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 11:25:01.678450 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673142    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 11:25:01.678450 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673144    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 11:25:01.678450 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673146    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 11:25:01.678450 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673149    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 11:25:01.678450 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673158    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 11:25:01.678450 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673161    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 11:25:01.678450 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673164    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 11:25:01.678450 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673167    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 11:25:01.678450 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673170    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 11:25:01.678450 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673173    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 11:25:01.678450 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673175    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 11:25:01.678450 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673178    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 11:25:01.678450 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673180    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 11:25:01.678450 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.673186    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 11:25:01.678450 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673329    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 11:25:01.678814 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673333    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 11:25:01.678814 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673336    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 11:25:01.678814 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673339    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 11:25:01.678814 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673341    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 11:25:01.678814 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673344    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 11:25:01.678814 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673347    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 11:25:01.678814 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673349    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 11:25:01.678814 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673352    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 11:25:01.678814 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673354    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 11:25:01.678814 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673357    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 11:25:01.678814 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673360    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 11:25:01.678814 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673362    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 11:25:01.678814 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673365    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 11:25:01.678814 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673368    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 11:25:01.678814 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673370    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 11:25:01.678814 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673373    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 11:25:01.678814 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673376    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 11:25:01.678814 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673378    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 11:25:01.678814 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673381    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 11:25:01.678814 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673383    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 11:25:01.679318 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673386    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 11:25:01.679318 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673389    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 11:25:01.679318 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673397    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 11:25:01.679318 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673400    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 11:25:01.679318 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673403    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 11:25:01.679318 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673407    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 11:25:01.679318 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673411    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 11:25:01.679318 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673415    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 11:25:01.679318 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673418    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 11:25:01.679318 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673421    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 11:25:01.679318 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673424    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 11:25:01.679318 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673426    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 11:25:01.679318 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673429    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 11:25:01.679318 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673432    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 11:25:01.679318 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673434    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 11:25:01.679318 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673437    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 11:25:01.679318 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673439    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 11:25:01.679318 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673442    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 11:25:01.679318 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673445    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 11:25:01.679778 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673447    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 11:25:01.679778 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673450    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 11:25:01.679778 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673453    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 11:25:01.679778 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673455    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 11:25:01.679778 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673458    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 11:25:01.679778 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673460    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 11:25:01.679778 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673463    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 11:25:01.679778 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673465    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 11:25:01.679778 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673468    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 11:25:01.679778 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673470    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 11:25:01.679778 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673479    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 11:25:01.679778 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673481    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 11:25:01.679778 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673484    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 11:25:01.679778 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673487    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 11:25:01.679778 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673489    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 11:25:01.679778 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673492    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 11:25:01.679778 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673501    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 11:25:01.679778 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673504    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 11:25:01.679778 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673507    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 11:25:01.679778 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673510    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 11:25:01.680292 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673512    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 11:25:01.680292 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673515    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 11:25:01.680292 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673517    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 11:25:01.680292 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673520    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 11:25:01.680292 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673523    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 11:25:01.680292 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673525    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 11:25:01.680292 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673528    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 11:25:01.680292 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673531    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 11:25:01.680292 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673534    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 11:25:01.680292 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673536    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 11:25:01.680292 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673539    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 11:25:01.680292 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673541    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 11:25:01.680292 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673544    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 11:25:01.680292 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673546    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 11:25:01.680292 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673549    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 11:25:01.680292 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673551    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 11:25:01.680292 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673554    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 11:25:01.680292 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673556    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 11:25:01.680292 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673559    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 11:25:01.680800 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673561    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 11:25:01.680800 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673564    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 11:25:01.680800 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673566    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 11:25:01.680800 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673569    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 11:25:01.680800 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673572    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 11:25:01.680800 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673575    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 11:25:01.680800 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673577    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 11:25:01.680800 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673580    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 11:25:01.680800 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673582    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 11:25:01.680800 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673585    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 11:25:01.680800 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673593    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 11:25:01.680800 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673597    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 11:25:01.680800 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673599    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 11:25:01.680800 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673601    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 11:25:01.680800 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673605    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 11:25:01.680800 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673609    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 11:25:01.680800 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673612    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 11:25:01.680800 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673615    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 11:25:01.680800 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673618    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 11:25:01.681274 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:01.673620    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 11:25:01.681274 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.673626    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 11:25:01.681274 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.673726    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 11:25:01.681274 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.676166    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 11:25:01.681274 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.677311    2562 server.go:1019] "Starting client certificate rotation"
Sep 15 11:25:01.681274 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.677415    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 11:25:01.681274 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.677443    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 11:25:01.681274 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.680534    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 11:25:01.681993 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.681975    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 11:25:01.698372 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.698346    2562 log.go:25] "Validated CRI v1 runtime API"
Sep 15 11:25:01.701919 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.701903    2562 log.go:25] "Validated CRI v1 image API"
Sep 15 11:25:01.703621 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.703606    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 11:25:01.706141 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.706119    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 11:25:01.709030 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.709011    2562 fs.go:135] Filesystem UUIDs: map[4c39e92f-f3fb-43b5-8426-5149ab3b2f40:/dev/nvme0n1p3 7962ff52-3b94-4d42-859c-df5d34b820c0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 15 11:25:01.709093 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.709031    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 11:25:01.715733 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.715620    2562 manager.go:216] Machine: {Timestamp:2026-09-15 11:25:01.713143475 +0000 UTC m=+0.375638668 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099215 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f27bedbd140817c67ba4124f248b9 SystemUUID:ec2f27be-dbd1-4081-7c67-ba4124f248b9 BootID:08438492-61b6-4dd1-98d6-ffa902dfd534 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c8:cd:13:53 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c8:cd:13:53 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:46:49:20:5b:de Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 11:25:01.715733 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.715731    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 11:25:01.715842 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.715816    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 11:25:01.717861 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.717837    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 11:25:01.718018 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.717864    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-143.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 11:25:01.719526 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.719516    2562 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 11:25:01.719565 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.719528    2562 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 11:25:01.719565 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.719536    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 11:25:01.719565 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.719551    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 11:25:01.720788 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.720778    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 11:25:01.725163 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.725152    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 11:25:01.725277 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.725268    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 11:25:01.725314 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.725307    2562 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 11:25:01.725343 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.725319    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 11:25:01.725343 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.725341    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 11:25:01.725393 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.725350    2562 kubelet.go:398] "Adding apiserver pod source"
Sep 15 11:25:01.725393 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.725367    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 11:25:01.725756 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.725744    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 11:25:01.725798 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.725765    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 11:25:01.728268 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.728229    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f2mbz"
Sep 15 11:25:01.729307 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.729289    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 11:25:01.731886 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.731869    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 11:25:01.731963 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.731901    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 11:25:01.733581 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.733561    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f2mbz"
Sep 15 11:25:01.733936 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.733923    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 11:25:01.733992 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.733940    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 11:25:01.733992 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.733946    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 11:25:01.733992 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.733951    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 11:25:01.733992 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.733959    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 11:25:01.733992 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.733965    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 11:25:01.733992 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.733978    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 11:25:01.733992 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.733983    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 11:25:01.733992 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.733990    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 11:25:01.733992 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.733996    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 11:25:01.734255 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.734005    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 11:25:01.734255 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.734010    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 11:25:01.735138 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:01.735119    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-143.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 11:25:01.735321 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:01.735300    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 11:25:01.735720 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.735707    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 11:25:01.735758 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.735723    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 11:25:01.739724 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.739710    2562 server.go:1268] "Started kubelet"
Sep 15 11:25:01.740347 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.740305    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 11:25:01.740434 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.740379    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 11:25:01.740602 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.740566    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 11:25:01.742445 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.742245    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 11:25:01.742582 ip-10-0-135-143 systemd[1]: Started Kubernetes Kubelet.
Sep 15 11:25:01.744454 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.744333    2562 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 11:25:01.746363 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:01.746342    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 11:25:01.755404 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.755384    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 11:25:01.756139 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.756119    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 11:25:01.756382 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:01.756362    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Sep 15 11:25:01.756962 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.756944    2562 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-143.ec2.internal" not found
Sep 15 11:25:01.757085 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.757010    2562 factory.go:55] Registering systemd factory
Sep 15 11:25:01.757085 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.757025    2562 factory.go:223] Registration of the systemd container factory successfully
Sep 15 11:25:01.757085 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.757024    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 11:25:01.757085 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.757047    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 11:25:01.757085 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.757023    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 11:25:01.757332 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.757144    2562 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 11:25:01.757332 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.757152    2562 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 11:25:01.757332 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.757320    2562 factory.go:153] Registering CRI-O factory
Sep 15 11:25:01.757332 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.757331    2562 factory.go:223] Registration of the crio container factory successfully
Sep 15 11:25:01.757526 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.757389    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 11:25:01.757526 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.757402    2562 factory.go:103] Registering Raw factory
Sep 15 11:25:01.757526 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.757415    2562 manager.go:1196] Started watching for new ooms in manager
Sep 15 11:25:01.758518 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.758494    2562 manager.go:349] Starting recovery of all containers
Sep 15 11:25:01.759442 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.759410    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 11:25:01.765680 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:01.765660    2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-143.ec2.internal\" not found" node="ip-10-0-135-143.ec2.internal"
Sep 15 11:25:01.768862 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.768846    2562 manager.go:354] Recovery completed
Sep 15 11:25:01.771871 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.771851    2562 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-143.ec2.internal" not found
Sep 15 11:25:01.773287 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.773275    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:25:01.775327 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.775308    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:25:01.775416 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.775337    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:25:01.775416 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.775347    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:25:01.775892 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.775880    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 11:25:01.775955 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.775892    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:25:01.775955 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.775894    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 11:25:01.775955 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.775915    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 11:25:01.776634 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.776621    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:25:01.776690 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.776643    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:25:01.776690 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.776652    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:25:01.777963 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.777951    2562 policy_none.go:49] "None policy: Start"
Sep 15 11:25:01.778011 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.777967    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 11:25:01.778011 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.777976    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 11:25:01.779778 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.779766    2562 policy_none.go:47] "Start"
Sep 15 11:25:01.844389 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.824968    2562 manager.go:337] "Starting Device Plugin manager"
Sep 15 11:25:01.844389 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:01.825027    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 11:25:01.844389 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.825041    2562 server.go:87] "Starting device plugin registration server"
Sep 15 11:25:01.844389 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.825387    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 11:25:01.844389 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.825402    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 11:25:01.844389 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.825571    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 11:25:01.844389 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.825656    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 11:25:01.844389 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.825664    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 11:25:01.844389 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:01.826265    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 11:25:01.844389 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:01.826322    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-143.ec2.internal\" not found"
Sep 15 11:25:01.844389 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.828920    2562 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-143.ec2.internal" not found
Sep 15 11:25:01.865801 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.865771    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 11:25:01.867011 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.866995    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 11:25:01.867078 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.867043    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 11:25:01.867126 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.867079    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 11:25:01.867126 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:01.867122    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 11:25:01.868961 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.868944    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 11:25:01.926047 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.925843    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:25:01.927017 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.926997    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:25:01.927089 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.927032    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:25:01.927089 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.927043    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:25:01.927153 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.927112    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-143.ec2.internal"
Sep 15 11:25:01.936105 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.936090    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-143.ec2.internal"
Sep 15 11:25:01.936163 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:01.936113    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-143.ec2.internal\": node \"ip-10-0-135-143.ec2.internal\" not found"
Sep 15 11:25:01.952927 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:01.952904    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Sep 15 11:25:01.967341 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.967317    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"]
Sep 15 11:25:01.967397 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.967389    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:25:01.968322 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.968308    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:25:01.968397 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.968340    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:25:01.968397 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.968350    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:25:01.970581 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.970569    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:25:01.970697 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.970684    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Sep 15 11:25:01.970736 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.970712    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:25:01.971309 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.971290    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:25:01.971425 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.971290    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:25:01.971425 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.971347    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:25:01.971425 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.971363    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:25:01.971425 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.971318    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:25:01.971425 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.971398    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:25:01.973597 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.973583    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"
Sep 15 11:25:01.973668 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.973607    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:25:01.974276 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.974260    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:25:01.974354 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.974291    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:25:01.974354 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:01.974307    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:25:01.996617 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:01.996598    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-143.ec2.internal\" not found" node="ip-10-0-135-143.ec2.internal"
Sep 15 11:25:02.000567 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:02.000551    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-143.ec2.internal\" not found" node="ip-10-0-135-143.ec2.internal"
Sep 15 11:25:02.054040 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:02.054007    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Sep 15 11:25:02.058353 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:02.058329    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2a68a194c773c52a06b82b6c37030522-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"2a68a194c773c52a06b82b6c37030522\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Sep 15 11:25:02.058469 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:02.058364    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a68a194c773c52a06b82b6c37030522-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"2a68a194c773c52a06b82b6c37030522\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Sep 15 11:25:02.058469 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:02.058389    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4fc673a9ff7f802e0d691cff004f3253-config\") pod \"kube-apiserver-proxy-ip-10-0-135-143.ec2.internal\" (UID: \"4fc673a9ff7f802e0d691cff004f3253\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"
Sep 15 11:25:02.154476 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:02.154432    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Sep 15 11:25:02.158710 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:02.158689    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2a68a194c773c52a06b82b6c37030522-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"2a68a194c773c52a06b82b6c37030522\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Sep 15 11:25:02.158757 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:02.158721    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a68a194c773c52a06b82b6c37030522-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"2a68a194c773c52a06b82b6c37030522\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Sep 15 11:25:02.158757 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:02.158739    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4fc673a9ff7f802e0d691cff004f3253-config\") pod \"kube-apiserver-proxy-ip-10-0-135-143.ec2.internal\" (UID: \"4fc673a9ff7f802e0d691cff004f3253\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"
Sep 15 11:25:02.158859 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:02.158806    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4fc673a9ff7f802e0d691cff004f3253-config\") pod \"kube-apiserver-proxy-ip-10-0-135-143.ec2.internal\" (UID: \"4fc673a9ff7f802e0d691cff004f3253\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"
Sep 15 11:25:02.158859 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:02.158808    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2a68a194c773c52a06b82b6c37030522-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"2a68a194c773c52a06b82b6c37030522\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Sep 15 11:25:02.158859 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:02.158835    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a68a194c773c52a06b82b6c37030522-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"2a68a194c773c52a06b82b6c37030522\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Sep 15 11:25:02.255207 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:02.255135    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Sep 15 11:25:02.298616 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:02.298585    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Sep 15 11:25:02.303470 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:02.303216    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"
Sep 15 11:25:02.356547 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:02.356500    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Sep 15 11:25:02.457629 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:02.457595    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Sep 15 11:25:02.562771 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:02.562699    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Sep 15 11:25:02.663181 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:02.663144    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Sep 15 11:25:02.678065 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:02.678022    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 11:25:02.678241 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:02.678220    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 11:25:02.678294 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:02.678259    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 11:25:02.740105 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:02.740017    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 11:20:02 +0000 UTC" deadline="2028-03-23 07:36:28.554986646 +0000 UTC"
Sep 15 11:25:02.740105 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:02.740097    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13316h11m25.814893354s"
Sep 15 11:25:02.763645 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:02.763605    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Sep 15 11:25:02.803699 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:02.803663    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 11:25:02.864075 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:02.863979    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Sep 15 11:25:02.965175 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:02.965126    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Sep 15 11:25:03.066121 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:03.066080    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Sep 15 11:25:03.100304 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.100273    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 11:25:03.160597 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.160506    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"
Sep 15 11:25:03.170070 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.170034    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 11:25:03.171022 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.171008    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Sep 15 11:25:03.184666 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.184648    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 11:25:03.503905 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.503821    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 11:25:03.710248 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.710215    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 11:25:03.727886 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.727861    2562 apiserver.go:52] "Watching apiserver"
Sep 15 11:25:03.735390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.734390    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 11:25:03.735390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.735294    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-b8qtl","kube-system/global-pull-secret-syncer-7j9vh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8","openshift-cluster-node-tuning-operator/tuned-mfm26","openshift-image-registry/node-ca-42tpl","openshift-multus/multus-additional-cni-plugins-qk259","openshift-network-operator/iptables-alerter-qw6lz","kube-system/konnectivity-agent-sr9gc","kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal","openshift-dns/node-resolver-f5l6r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal","openshift-multus/multus-v8qdz","openshift-multus/network-metrics-daemon-cc7mk","openshift-network-diagnostics/network-check-target-4gbpc"]
Sep 15 11:25:03.738495 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.738473    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qw6lz"
Sep 15 11:25:03.740597 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.740574    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 11:25:03.740729 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.740709    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:03.740818 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:03.740794    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc7mk" podUID="355f2000-c95f-4d71-b450-03ce72b5ea62"
Sep 15 11:25:03.740874 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.740852    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jhvzr\""
Sep 15 11:25:03.741020 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.741000    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 11:25:03.741147 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.741132    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 11:25:03.743033 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.743013    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.744801 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.744778    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 11:25:03.744901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.744865    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 11:25:03.745117 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.745100    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 11:25:03.745193 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.745100    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-77gk9\""
Sep 15 11:25:03.745365 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.745349    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.747279 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.747260    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8xqtk\""
Sep 15 11:25:03.747369 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.747282    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 11:25:03.747369 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.747260    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 11:25:03.747683 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.747652    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-42tpl"
Sep 15 11:25:03.749376 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.749352    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 11:25:03.749484 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.749425    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 11:25:03.749561 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.749483    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fl5hm\""
Sep 15 11:25:03.749717 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.749701    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 11:25:03.750087 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.750072    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.751752 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.751735    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 11:25:03.751851 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.751800    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 11:25:03.751914 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.751879    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ss95z\""
Sep 15 11:25:03.752054 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.752038    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 11:25:03.752239 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.752218    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 11:25:03.752297 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.752219    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 11:25:03.752465 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.752451    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:03.752559 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:03.752529    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gbpc" podUID="a0d6a68c-1fa6-4f31-b6f8-3a1a38988836"
Sep 15 11:25:03.755036 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.754988    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.756125 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.756105    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 11:25:03.756911 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.756895    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m958l\""
Sep 15 11:25:03.757694 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.757506    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:03.757694 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.757536    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 11:25:03.757694 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.757551    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 11:25:03.757694 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.757565    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 11:25:03.757694 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:03.757564    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7j9vh" podUID="85b6a362-7f90-4e54-89d2-2c21ef687efb"
Sep 15 11:25:03.757694 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.757542    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 11:25:03.757694 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.757598    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 11:25:03.757694 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.757624    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 11:25:03.760006 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.759979    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f5l6r"
Sep 15 11:25:03.761995 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.761969    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 11:25:03.762181 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.762094    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 11:25:03.762534 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.762515    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qbmqv\""
Sep 15 11:25:03.762812 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.762796    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sr9gc"
Sep 15 11:25:03.764905 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.764641    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sv5sv\""
Sep 15 11:25:03.764905 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.764749    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 11:25:03.764905 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.764775    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 11:25:03.765134 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.764942    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.767478 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.766940    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 11:25:03.767478 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.767244    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-47zqv\""
Sep 15 11:25:03.767688 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.767547    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 11:25:03.770279 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770256    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kppkp\" (UniqueName: \"kubernetes.io/projected/355f2000-c95f-4d71-b450-03ce72b5ea62-kube-api-access-kppkp\") pod \"network-metrics-daemon-cc7mk\" (UID: \"355f2000-c95f-4d71-b450-03ce72b5ea62\") " pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:03.770375 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770281    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-run-systemd\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.770375 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770299    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5d28bab-39d1-4b87-bf7d-93dea93300f3-sys-fs\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.770375 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770315    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-run\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.770375 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770332    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8nlr\" (UniqueName: \"kubernetes.io/projected/38e8a164-314f-4cfd-8121-edea9ed65041-kube-api-access-m8nlr\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.770375 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770355    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-run-netns\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.770375 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770373    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.770573 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770416    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5d28bab-39d1-4b87-bf7d-93dea93300f3-device-dir\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.770573 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770442    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/38e8a164-314f-4cfd-8121-edea9ed65041-etc-tuned\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.770573 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770462    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.770573 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770478    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-slash\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.770573 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770493    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-etc-systemd\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.770573 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770519    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm98q\" (UniqueName: \"kubernetes.io/projected/dbb4a488-5113-44e5-bea0-b831896d7cac-kube-api-access-sm98q\") pod \"node-resolver-f5l6r\" (UID: \"dbb4a488-5113-44e5-bea0-b831896d7cac\") " pod="openshift-dns/node-resolver-f5l6r"
Sep 15 11:25:03.770573 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770540    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-ovnkube-config\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.770573 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770555    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-ovnkube-script-lib\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.770573 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770569    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-etc-sysctl-d\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.770857 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770583    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-etc-sysconfig\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.770857 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770616    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dphg9\" (UniqueName: \"kubernetes.io/projected/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-kube-api-access-dphg9\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.770857 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770648    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-run-ovn\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.770857 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770664    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-cni-netd\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.770857 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770694    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-etc-modprobe-d\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.770857 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770711    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-etc-kubernetes\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.770857 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770733    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsms7\" (UniqueName: \"kubernetes.io/projected/515a11a1-5808-486b-b178-a3cf92c663d8-kube-api-access-wsms7\") pod \"iptables-alerter-qw6lz\" (UID: \"515a11a1-5808-486b-b178-a3cf92c663d8\") " pod="openshift-network-operator/iptables-alerter-qw6lz"
Sep 15 11:25:03.770857 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770764    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-cni-bin\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.770857 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770790    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-lib-modules\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.770857 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770807    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqwkz\" (UniqueName: \"kubernetes.io/projected/85bb9ddc-8249-4923-a96d-4b17b5450c51-kube-api-access-qqwkz\") pod \"node-ca-42tpl\" (UID: \"85bb9ddc-8249-4923-a96d-4b17b5450c51\") " pod="openshift-image-registry/node-ca-42tpl"
Sep 15 11:25:03.770857 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770827    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-system-cni-dir\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.770857 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770844    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/515a11a1-5808-486b-b178-a3cf92c663d8-host-slash\") pod \"iptables-alerter-qw6lz\" (UID: \"515a11a1-5808-486b-b178-a3cf92c663d8\") " pod="openshift-network-operator/iptables-alerter-qw6lz"
Sep 15 11:25:03.770857 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770858    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-run-ovn-kubernetes\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.771508 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770873    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs\") pod \"network-metrics-daemon-cc7mk\" (UID: \"355f2000-c95f-4d71-b450-03ce72b5ea62\") " pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:03.771508 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770887    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5d28bab-39d1-4b87-bf7d-93dea93300f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.771508 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770901    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/515a11a1-5808-486b-b178-a3cf92c663d8-iptables-alerter-script\") pod \"iptables-alerter-qw6lz\" (UID: \"515a11a1-5808-486b-b178-a3cf92c663d8\") " pod="openshift-network-operator/iptables-alerter-qw6lz"
Sep 15 11:25:03.771508 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770926    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-kubelet\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.771508 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.770966    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-node-log\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.771508 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771004    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-log-socket\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.771508 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771033    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5d28bab-39d1-4b87-bf7d-93dea93300f3-socket-dir\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.771508 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771060    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.771508 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771106    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-var-lib-openvswitch\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.771508 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771133    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-etc-openvswitch\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.771508 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771156    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbb4a488-5113-44e5-bea0-b831896d7cac-tmp-dir\") pod \"node-resolver-f5l6r\" (UID: \"dbb4a488-5113-44e5-bea0-b831896d7cac\") " pod="openshift-dns/node-resolver-f5l6r"
Sep 15 11:25:03.771508 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771179    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/85b6a362-7f90-4e54-89d2-2c21ef687efb-kubelet-config\") pod \"global-pull-secret-syncer-7j9vh\" (UID: \"85b6a362-7f90-4e54-89d2-2c21ef687efb\") " pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:03.771508 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771203    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38e8a164-314f-4cfd-8121-edea9ed65041-tmp\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.771508 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771226    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85bb9ddc-8249-4923-a96d-4b17b5450c51-serviceca\") pod \"node-ca-42tpl\" (UID: \"85bb9ddc-8249-4923-a96d-4b17b5450c51\") " pod="openshift-image-registry/node-ca-42tpl"
Sep 15 11:25:03.771508 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771252    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-cnibin\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.771508 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771276    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-ovn-node-metrics-cert\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.772077 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771298    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-sys\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.772077 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771320    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85bb9ddc-8249-4923-a96d-4b17b5450c51-host\") pod \"node-ca-42tpl\" (UID: \"85bb9ddc-8249-4923-a96d-4b17b5450c51\") " pod="openshift-image-registry/node-ca-42tpl"
Sep 15 11:25:03.772077 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771343    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/85b6a362-7f90-4e54-89d2-2c21ef687efb-dbus\") pod \"global-pull-secret-syncer-7j9vh\" (UID: \"85b6a362-7f90-4e54-89d2-2c21ef687efb\") " pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:03.772077 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret\") pod \"global-pull-secret-syncer-7j9vh\" (UID: \"85b6a362-7f90-4e54-89d2-2c21ef687efb\") " pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:03.772077 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771395    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-var-lib-kubelet\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.772077 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771418    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-systemd-units\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.772077 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771444    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-run-openvswitch\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.772077 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771468    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5d28bab-39d1-4b87-bf7d-93dea93300f3-registration-dir\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.772077 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771496    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6qkl\" (UniqueName: \"kubernetes.io/projected/f5d28bab-39d1-4b87-bf7d-93dea93300f3-kube-api-access-p6qkl\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.772077 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771519    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-cni-binary-copy\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.772077 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771545    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dbb4a488-5113-44e5-bea0-b831896d7cac-hosts-file\") pod \"node-resolver-f5l6r\" (UID: \"dbb4a488-5113-44e5-bea0-b831896d7cac\") " pod="openshift-dns/node-resolver-f5l6r"
Sep 15 11:25:03.772077 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771571    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-env-overrides\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.772077 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771595    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52l4s\" (UniqueName: \"kubernetes.io/projected/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-kube-api-access-52l4s\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.772077 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771620    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5d28bab-39d1-4b87-bf7d-93dea93300f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.772077 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771644    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-etc-sysctl-conf\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.772077 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771667    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-host\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.772608 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771689    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.772608 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.771716    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q6gd\" (UniqueName: \"kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd\") pod \"network-check-target-4gbpc\" (UID: \"a0d6a68c-1fa6-4f31-b6f8-3a1a38988836\") " pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:03.790761 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.790743    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zd9r7"
Sep 15 11:25:03.798743 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.798723    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zd9r7"
Sep 15 11:25:03.858232 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.858208    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 11:25:03.872627 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.872589    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5d28bab-39d1-4b87-bf7d-93dea93300f3-device-dir\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.872770 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.872631    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/38e8a164-314f-4cfd-8121-edea9ed65041-etc-tuned\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.872770 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.872668    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.872770 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.872679    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5d28bab-39d1-4b87-bf7d-93dea93300f3-device-dir\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.872770 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.872693    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-slash\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.872770 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.872725    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-multus-cni-dir\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.873046 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.872749    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-etc-systemd\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.873046 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.872786    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-slash\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.873046 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.872800    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm98q\" (UniqueName: \"kubernetes.io/projected/dbb4a488-5113-44e5-bea0-b831896d7cac-kube-api-access-sm98q\") pod \"node-resolver-f5l6r\" (UID: \"dbb4a488-5113-44e5-bea0-b831896d7cac\") " pod="openshift-dns/node-resolver-f5l6r"
Sep 15 11:25:03.873046 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.872840    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-ovnkube-config\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.873046 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.872855    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.873046 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.872870    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-ovnkube-script-lib\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.873046 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.872877    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-etc-systemd\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.873046 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.872898    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-etc-sysctl-d\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.873046 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.872924    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-etc-sysconfig\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.873046 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.872954    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dphg9\" (UniqueName: \"kubernetes.io/projected/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-kube-api-access-dphg9\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.873046 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.872979    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-run-ovn\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.873046 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.872984    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 11:25:03.873046 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873006    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-cni-netd\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.873046 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873049    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-cni-netd\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.873793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873094    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-etc-sysctl-d\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.873793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873107    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-run-ovn\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.873793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873172    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-etc-sysconfig\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.873793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873212    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-etc-modprobe-d\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.873793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873258    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-etc-kubernetes\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.873793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873281    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsms7\" (UniqueName: \"kubernetes.io/projected/515a11a1-5808-486b-b178-a3cf92c663d8-kube-api-access-wsms7\") pod \"iptables-alerter-qw6lz\" (UID: \"515a11a1-5808-486b-b178-a3cf92c663d8\") " pod="openshift-network-operator/iptables-alerter-qw6lz"
Sep 15 11:25:03.873793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873297    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-cni-bin\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.873793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873322    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-host-run-k8s-cni-cncf-io\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.873793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873338    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-host-var-lib-cni-bin\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.873793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873357    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-lib-modules\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.873793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873364    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-etc-modprobe-d\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.873793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873374    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqwkz\" (UniqueName: \"kubernetes.io/projected/85bb9ddc-8249-4923-a96d-4b17b5450c51-kube-api-access-qqwkz\") pod \"node-ca-42tpl\" (UID: \"85bb9ddc-8249-4923-a96d-4b17b5450c51\") " pod="openshift-image-registry/node-ca-42tpl"
Sep 15 11:25:03.873793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873423    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-system-cni-dir\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.873793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873453    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/515a11a1-5808-486b-b178-a3cf92c663d8-host-slash\") pod \"iptables-alerter-qw6lz\" (UID: \"515a11a1-5808-486b-b178-a3cf92c663d8\") " pod="openshift-network-operator/iptables-alerter-qw6lz"
Sep 15 11:25:03.873793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873486    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-run-ovn-kubernetes\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.873793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873504    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-ovnkube-config\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.873793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873514    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs\") pod \"network-metrics-daemon-cc7mk\" (UID: \"355f2000-c95f-4d71-b450-03ce72b5ea62\") " pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:03.874706 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873546    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5d28bab-39d1-4b87-bf7d-93dea93300f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.874706 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873555    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-cni-bin\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.874706 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873572    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/515a11a1-5808-486b-b178-a3cf92c663d8-iptables-alerter-script\") pod \"iptables-alerter-qw6lz\" (UID: \"515a11a1-5808-486b-b178-a3cf92c663d8\") " pod="openshift-network-operator/iptables-alerter-qw6lz"
Sep 15 11:25:03.874706 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873612    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-lib-modules\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.874706 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873623    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-etc-kubernetes\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.874706 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873650    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/515a11a1-5808-486b-b178-a3cf92c663d8-host-slash\") pod \"iptables-alerter-qw6lz\" (UID: \"515a11a1-5808-486b-b178-a3cf92c663d8\") " pod="openshift-network-operator/iptables-alerter-qw6lz"
Sep 15 11:25:03.874706 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873664    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-kubelet\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.874706 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873681    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-system-cni-dir\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.874706 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873690    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-node-log\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.874706 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873714    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-log-socket\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.874706 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873741    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-system-cni-dir\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.874706 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873514    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-ovnkube-script-lib\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.874706 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873768    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-kubelet\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.874706 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873793    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5d28bab-39d1-4b87-bf7d-93dea93300f3-socket-dir\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.874706 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873808    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-run-ovn-kubernetes\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.874706 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873811    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-log-socket\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.874706 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:03.873744    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:03.874706 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873839    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-node-log\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.875390 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:03.873893    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs podName:355f2000-c95f-4d71-b450-03ce72b5ea62 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:04.373874021 +0000 UTC m=+3.036369204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs") pod "network-metrics-daemon-cc7mk" (UID: "355f2000-c95f-4d71-b450-03ce72b5ea62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:03.875390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873915    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5d28bab-39d1-4b87-bf7d-93dea93300f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.875390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874002    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5d28bab-39d1-4b87-bf7d-93dea93300f3-socket-dir\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.875390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.873822    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.875390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874037    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-var-lib-openvswitch\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.875390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874060    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-etc-openvswitch\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.875390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874087    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-etc-kubernetes\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.875390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874091    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-var-lib-openvswitch\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.875390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874106    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbb4a488-5113-44e5-bea0-b831896d7cac-tmp-dir\") pod \"node-resolver-f5l6r\" (UID: \"dbb4a488-5113-44e5-bea0-b831896d7cac\") " pod="openshift-dns/node-resolver-f5l6r"
Sep 15 11:25:03.875390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874149    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-etc-openvswitch\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.875390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874109    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/515a11a1-5808-486b-b178-a3cf92c663d8-iptables-alerter-script\") pod \"iptables-alerter-qw6lz\" (UID: \"515a11a1-5808-486b-b178-a3cf92c663d8\") " pod="openshift-network-operator/iptables-alerter-qw6lz"
Sep 15 11:25:03.875390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874168    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/85b6a362-7f90-4e54-89d2-2c21ef687efb-kubelet-config\") pod \"global-pull-secret-syncer-7j9vh\" (UID: \"85b6a362-7f90-4e54-89d2-2c21ef687efb\") " pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:03.875390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874191    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-host-var-lib-cni-multus\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.875390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38e8a164-314f-4cfd-8121-edea9ed65041-tmp\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.875390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874225    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85bb9ddc-8249-4923-a96d-4b17b5450c51-serviceca\") pod \"node-ca-42tpl\" (UID: \"85bb9ddc-8249-4923-a96d-4b17b5450c51\") " pod="openshift-image-registry/node-ca-42tpl"
Sep 15 11:25:03.875390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874229    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/85b6a362-7f90-4e54-89d2-2c21ef687efb-kubelet-config\") pod \"global-pull-secret-syncer-7j9vh\" (UID: \"85b6a362-7f90-4e54-89d2-2c21ef687efb\") " pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:03.876267 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874242    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-cnibin\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.876267 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874279    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-cnibin\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.876267 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874281    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-ovn-node-metrics-cert\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.876267 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874316    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-sys\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.876267 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874345    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85bb9ddc-8249-4923-a96d-4b17b5450c51-host\") pod \"node-ca-42tpl\" (UID: \"85bb9ddc-8249-4923-a96d-4b17b5450c51\") " pod="openshift-image-registry/node-ca-42tpl"
Sep 15 11:25:03.876267 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874371    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/85b6a362-7f90-4e54-89d2-2c21ef687efb-dbus\") pod \"global-pull-secret-syncer-7j9vh\" (UID: \"85b6a362-7f90-4e54-89d2-2c21ef687efb\") " pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:03.876267 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874399    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret\") pod \"global-pull-secret-syncer-7j9vh\" (UID: \"85b6a362-7f90-4e54-89d2-2c21ef687efb\") " pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:03.876267 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874429    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c96758c8-f869-41dd-9350-e015e30b82df-cni-binary-copy\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.876267 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-var-lib-kubelet\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.876267 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874484    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbb4a488-5113-44e5-bea0-b831896d7cac-tmp-dir\") pod \"node-resolver-f5l6r\" (UID: \"dbb4a488-5113-44e5-bea0-b831896d7cac\") " pod="openshift-dns/node-resolver-f5l6r"
Sep 15 11:25:03.876267 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874487    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-systemd-units\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.876267 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874532    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-systemd-units\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.876267 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874366    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.876267 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874564    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-run-openvswitch\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.876267 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874594    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-cnibin\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.876267 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874617    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c96758c8-f869-41dd-9350-e015e30b82df-multus-daemon-config\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.876267 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:03.874625    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:03.876267 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874643    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5d28bab-39d1-4b87-bf7d-93dea93300f3-registration-dir\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.877157 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:03.874673    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret podName:85b6a362-7f90-4e54-89d2-2c21ef687efb nodeName:}" failed. No retries permitted until 2026-09-15 11:25:04.374654347 +0000 UTC m=+3.037149510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret") pod "global-pull-secret-syncer-7j9vh" (UID: "85b6a362-7f90-4e54-89d2-2c21ef687efb") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:03.877157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874692    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/85b6a362-7f90-4e54-89d2-2c21ef687efb-dbus\") pod \"global-pull-secret-syncer-7j9vh\" (UID: \"85b6a362-7f90-4e54-89d2-2c21ef687efb\") " pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:03.877157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874694    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6qkl\" (UniqueName: \"kubernetes.io/projected/f5d28bab-39d1-4b87-bf7d-93dea93300f3-kube-api-access-p6qkl\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.877157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874722    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85bb9ddc-8249-4923-a96d-4b17b5450c51-serviceca\") pod \"node-ca-42tpl\" (UID: \"85bb9ddc-8249-4923-a96d-4b17b5450c51\") " pod="openshift-image-registry/node-ca-42tpl"
Sep 15 11:25:03.877157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874699    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5d28bab-39d1-4b87-bf7d-93dea93300f3-registration-dir\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.877157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874748    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-cni-binary-copy\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.877157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874776    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dbb4a488-5113-44e5-bea0-b831896d7cac-hosts-file\") pod \"node-resolver-f5l6r\" (UID: \"dbb4a488-5113-44e5-bea0-b831896d7cac\") " pod="openshift-dns/node-resolver-f5l6r"
Sep 15 11:25:03.877157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874795    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85bb9ddc-8249-4923-a96d-4b17b5450c51-host\") pod \"node-ca-42tpl\" (UID: \"85bb9ddc-8249-4923-a96d-4b17b5450c51\") " pod="openshift-image-registry/node-ca-42tpl"
Sep 15 11:25:03.877157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874798    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-sys\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.877157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874805    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-env-overrides\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.877157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874847    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52l4s\" (UniqueName: \"kubernetes.io/projected/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-kube-api-access-52l4s\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.877157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874855    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dbb4a488-5113-44e5-bea0-b831896d7cac-hosts-file\") pod \"node-resolver-f5l6r\" (UID: \"dbb4a488-5113-44e5-bea0-b831896d7cac\") " pod="openshift-dns/node-resolver-f5l6r"
Sep 15 11:25:03.877157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.874938    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-run-openvswitch\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.877157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.875940    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-cni-binary-copy\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.877157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.876023    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-var-lib-kubelet\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.877965 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.877372    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/38e8a164-314f-4cfd-8121-edea9ed65041-etc-tuned\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.877965 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.877654    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-env-overrides\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.877965 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.877815    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5d28bab-39d1-4b87-bf7d-93dea93300f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.877965 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.877888    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-etc-sysctl-conf\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.877965 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.877921    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-host\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.877965 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.877945    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38e8a164-314f-4cfd-8121-edea9ed65041-tmp\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.877965 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.877956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.878328 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.877992    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-ovn-node-metrics-cert\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.878328 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.878036    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6gd\" (UniqueName: \"kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd\") pod \"network-check-target-4gbpc\" (UID: \"a0d6a68c-1fa6-4f31-b6f8-3a1a38988836\") " pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:03.879951 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.879223    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-host\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.881618 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.881396    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5d28bab-39d1-4b87-bf7d-93dea93300f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.881618 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.881534    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-etc-sysctl-conf\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.881969 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.881940    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dphg9\" (UniqueName: \"kubernetes.io/projected/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-kube-api-access-dphg9\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.882233 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.882213    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d662f7e1-efac-4ded-8c57-4aa36d0e84a0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qk259\" (UID: \"d662f7e1-efac-4ded-8c57-4aa36d0e84a0\") " pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:03.883199 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.882487    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kppkp\" (UniqueName: \"kubernetes.io/projected/355f2000-c95f-4d71-b450-03ce72b5ea62-kube-api-access-kppkp\") pod \"network-metrics-daemon-cc7mk\" (UID: \"355f2000-c95f-4d71-b450-03ce72b5ea62\") " pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:03.883199 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.882587    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/870be114-239c-4f62-8def-c3144e7cbeb0-agent-certs\") pod \"konnectivity-agent-sr9gc\" (UID: \"870be114-239c-4f62-8def-c3144e7cbeb0\") " pod="kube-system/konnectivity-agent-sr9gc"
Sep 15 11:25:03.883199 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.882628    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-host-run-netns\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.883199 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.882665    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-run-systemd\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.883199 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.882696    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-multus-socket-dir-parent\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.883199 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.882731    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-host-var-lib-kubelet\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.883199 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.882794    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-hostroot\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.883199 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.882795    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-run-systemd\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.883199 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.882840    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-multus-conf-dir\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.883199 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.882908    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-host-run-multus-certs\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.883199 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.882932    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqwkz\" (UniqueName: \"kubernetes.io/projected/85bb9ddc-8249-4923-a96d-4b17b5450c51-kube-api-access-qqwkz\") pod \"node-ca-42tpl\" (UID: \"85bb9ddc-8249-4923-a96d-4b17b5450c51\") " pod="openshift-image-registry/node-ca-42tpl"
Sep 15 11:25:03.883199 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.882967    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrc22\" (UniqueName: \"kubernetes.io/projected/c96758c8-f869-41dd-9350-e015e30b82df-kube-api-access-zrc22\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.883199 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.883025    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5d28bab-39d1-4b87-bf7d-93dea93300f3-sys-fs\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.883199 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.883125    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5d28bab-39d1-4b87-bf7d-93dea93300f3-sys-fs\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.883199 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.883165    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-run\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.883199 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.883207    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8nlr\" (UniqueName: \"kubernetes.io/projected/38e8a164-314f-4cfd-8121-edea9ed65041-kube-api-access-m8nlr\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.883852 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.883270    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-run-netns\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.883852 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.883281    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/38e8a164-314f-4cfd-8121-edea9ed65041-run\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.883852 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.883322    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.883852 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.883360    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/870be114-239c-4f62-8def-c3144e7cbeb0-konnectivity-ca\") pod \"konnectivity-agent-sr9gc\" (UID: \"870be114-239c-4f62-8def-c3144e7cbeb0\") " pod="kube-system/konnectivity-agent-sr9gc"
Sep 15 11:25:03.883852 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.883372    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm98q\" (UniqueName: \"kubernetes.io/projected/dbb4a488-5113-44e5-bea0-b831896d7cac-kube-api-access-sm98q\") pod \"node-resolver-f5l6r\" (UID: \"dbb4a488-5113-44e5-bea0-b831896d7cac\") " pod="openshift-dns/node-resolver-f5l6r"
Sep 15 11:25:03.883852 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.883422    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.883852 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.883444    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-host-run-netns\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.883852 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.883455    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6qkl\" (UniqueName: \"kubernetes.io/projected/f5d28bab-39d1-4b87-bf7d-93dea93300f3-kube-api-access-p6qkl\") pod \"aws-ebs-csi-driver-node-jcff8\" (UID: \"f5d28bab-39d1-4b87-bf7d-93dea93300f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:03.883852 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.883484    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsms7\" (UniqueName: \"kubernetes.io/projected/515a11a1-5808-486b-b178-a3cf92c663d8-kube-api-access-wsms7\") pod \"iptables-alerter-qw6lz\" (UID: \"515a11a1-5808-486b-b178-a3cf92c663d8\") " pod="openshift-network-operator/iptables-alerter-qw6lz"
Sep 15 11:25:03.884799 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.884764    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52l4s\" (UniqueName: \"kubernetes.io/projected/e311c2ba-68b7-47af-92c3-a8dac7d2d28b-kube-api-access-52l4s\") pod \"ovnkube-node-b8qtl\" (UID: \"e311c2ba-68b7-47af-92c3-a8dac7d2d28b\") " pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:03.886047 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:03.886028    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 11:25:03.886172 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:03.886051    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 11:25:03.886172 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:03.886065    2562 projected.go:196] Error preparing data for projected volume kube-api-access-6q6gd for pod openshift-network-diagnostics/network-check-target-4gbpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:03.886172 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:03.886125    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd podName:a0d6a68c-1fa6-4f31-b6f8-3a1a38988836 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:04.386108009 +0000 UTC m=+3.048603189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6q6gd" (UniqueName: "kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd") pod "network-check-target-4gbpc" (UID: "a0d6a68c-1fa6-4f31-b6f8-3a1a38988836") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:03.888371 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.888350    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kppkp\" (UniqueName: \"kubernetes.io/projected/355f2000-c95f-4d71-b450-03ce72b5ea62-kube-api-access-kppkp\") pod \"network-metrics-daemon-cc7mk\" (UID: \"355f2000-c95f-4d71-b450-03ce72b5ea62\") " pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:03.889632 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.889614    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8nlr\" (UniqueName: \"kubernetes.io/projected/38e8a164-314f-4cfd-8121-edea9ed65041-kube-api-access-m8nlr\") pod \"tuned-mfm26\" (UID: \"38e8a164-314f-4cfd-8121-edea9ed65041\") " pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:03.984049 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984018    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-multus-socket-dir-parent\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984049 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984048    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-host-var-lib-kubelet\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984274 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984086    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-hostroot\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984274 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984113    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-multus-conf-dir\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984274 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984154    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-host-var-lib-kubelet\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984274 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984156    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-multus-socket-dir-parent\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984274 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984186    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-host-run-multus-certs\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984274 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984187    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-hostroot\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984274 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984223    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrc22\" (UniqueName: \"kubernetes.io/projected/c96758c8-f869-41dd-9350-e015e30b82df-kube-api-access-zrc22\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984274 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984226    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-host-run-multus-certs\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984274 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984222    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-multus-conf-dir\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984274 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984259    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/870be114-239c-4f62-8def-c3144e7cbeb0-konnectivity-ca\") pod \"konnectivity-agent-sr9gc\" (UID: \"870be114-239c-4f62-8def-c3144e7cbeb0\") " pod="kube-system/konnectivity-agent-sr9gc"
Sep 15 11:25:03.984766 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984300    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-multus-cni-dir\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984766 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984339    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-host-run-k8s-cni-cncf-io\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984766 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984370    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-host-run-k8s-cni-cncf-io\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984766 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984401    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-multus-cni-dir\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984766 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984427    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-host-var-lib-cni-bin\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984766 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984467    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-system-cni-dir\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984766 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984497    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-etc-kubernetes\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984766 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984514    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-host-var-lib-cni-multus\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984766 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984561    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c96758c8-f869-41dd-9350-e015e30b82df-cni-binary-copy\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984766 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984566    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-system-cni-dir\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984766 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984589    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-cnibin\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984766 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984566    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-etc-kubernetes\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984766 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984612    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-host-var-lib-cni-multus\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984766 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984618    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-host-var-lib-cni-bin\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984766 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984643    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-cnibin\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984766 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984620    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c96758c8-f869-41dd-9350-e015e30b82df-multus-daemon-config\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.984766 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984689    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/870be114-239c-4f62-8def-c3144e7cbeb0-agent-certs\") pod \"konnectivity-agent-sr9gc\" (UID: \"870be114-239c-4f62-8def-c3144e7cbeb0\") " pod="kube-system/konnectivity-agent-sr9gc"
Sep 15 11:25:03.984766 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984712    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-host-run-netns\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.985419 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984755    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c96758c8-f869-41dd-9350-e015e30b82df-host-run-netns\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.985419 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.984937    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/870be114-239c-4f62-8def-c3144e7cbeb0-konnectivity-ca\") pod \"konnectivity-agent-sr9gc\" (UID: \"870be114-239c-4f62-8def-c3144e7cbeb0\") " pod="kube-system/konnectivity-agent-sr9gc"
Sep 15 11:25:03.985419 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.985177    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c96758c8-f869-41dd-9350-e015e30b82df-cni-binary-copy\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.985419 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.985207    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c96758c8-f869-41dd-9350-e015e30b82df-multus-daemon-config\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:03.987239 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.987220    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/870be114-239c-4f62-8def-c3144e7cbeb0-agent-certs\") pod \"konnectivity-agent-sr9gc\" (UID: \"870be114-239c-4f62-8def-c3144e7cbeb0\") " pod="kube-system/konnectivity-agent-sr9gc"
Sep 15 11:25:03.991562 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:03.991537    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrc22\" (UniqueName: \"kubernetes.io/projected/c96758c8-f869-41dd-9350-e015e30b82df-kube-api-access-zrc22\") pod \"multus-v8qdz\" (UID: \"c96758c8-f869-41dd-9350-e015e30b82df\") " pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:04.069931 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.069851    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qw6lz"
Sep 15 11:25:04.095445 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.095414    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8"
Sep 15 11:25:04.101398 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.101372    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mfm26"
Sep 15 11:25:04.129078 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.129037    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-42tpl"
Sep 15 11:25:04.141386 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.141366    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qk259"
Sep 15 11:25:04.158366 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.158338    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:04.175412 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.175390    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f5l6r"
Sep 15 11:25:04.177944 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:04.177894    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd662f7e1_efac_4ded_8c57_4aa36d0e84a0.slice/crio-54a9a1a9603f31e9ec179728ba9b91c8d5346345fd4af53e7f2808f44e08af78 WatchSource:0}: Error finding container 54a9a1a9603f31e9ec179728ba9b91c8d5346345fd4af53e7f2808f44e08af78: Status 404 returned error can't find the container with id 54a9a1a9603f31e9ec179728ba9b91c8d5346345fd4af53e7f2808f44e08af78
Sep 15 11:25:04.178464 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:04.178442    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode311c2ba_68b7_47af_92c3_a8dac7d2d28b.slice/crio-4792228d3643dd04792310a8bb96b822d9c46392d4e90eee84a576767895b5e0 WatchSource:0}: Error finding container 4792228d3643dd04792310a8bb96b822d9c46392d4e90eee84a576767895b5e0: Status 404 returned error can't find the container with id 4792228d3643dd04792310a8bb96b822d9c46392d4e90eee84a576767895b5e0
Sep 15 11:25:04.179231 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:04.179201    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85bb9ddc_8249_4923_a96d_4b17b5450c51.slice/crio-0390ea7c6f91b2842b64f21fa76c622dd3a2806b22b83d33ddf217ad4b3680af WatchSource:0}: Error finding container 0390ea7c6f91b2842b64f21fa76c622dd3a2806b22b83d33ddf217ad4b3680af: Status 404 returned error can't find the container with id 0390ea7c6f91b2842b64f21fa76c622dd3a2806b22b83d33ddf217ad4b3680af
Sep 15 11:25:04.179879 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:04.179831    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a68a194c773c52a06b82b6c37030522.slice/crio-f7d050542c1542422f05c25cf24ec2b430c786081c2d859c461e861c18c40f11 WatchSource:0}: Error finding container f7d050542c1542422f05c25cf24ec2b430c786081c2d859c461e861c18c40f11: Status 404 returned error can't find the container with id f7d050542c1542422f05c25cf24ec2b430c786081c2d859c461e861c18c40f11
Sep 15 11:25:04.183279 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.183224    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 11:25:04.187021 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:04.186994    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb4a488_5113_44e5_bea0_b831896d7cac.slice/crio-65bf81bc3fd1353265ce2f279246156bc132c8a368981a62c4325cde69912de7 WatchSource:0}: Error finding container 65bf81bc3fd1353265ce2f279246156bc132c8a368981a62c4325cde69912de7: Status 404 returned error can't find the container with id 65bf81bc3fd1353265ce2f279246156bc132c8a368981a62c4325cde69912de7
Sep 15 11:25:04.209552 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.209526    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sr9gc"
Sep 15 11:25:04.214616 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.214594    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v8qdz"
Sep 15 11:25:04.230605 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:04.230580    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870be114_239c_4f62_8def_c3144e7cbeb0.slice/crio-eac4cbdc24d52a177bb5c35fce931451c61ecf488ed24e43c8b16ef3fa251baf WatchSource:0}: Error finding container eac4cbdc24d52a177bb5c35fce931451c61ecf488ed24e43c8b16ef3fa251baf: Status 404 returned error can't find the container with id eac4cbdc24d52a177bb5c35fce931451c61ecf488ed24e43c8b16ef3fa251baf
Sep 15 11:25:04.231243 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:04.231218    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc96758c8_f869_41dd_9350_e015e30b82df.slice/crio-4198631295aced9fb4273be00a2162f9a0a741538a6941b964830afefba5434b WatchSource:0}: Error finding container 4198631295aced9fb4273be00a2162f9a0a741538a6941b964830afefba5434b: Status 404 returned error can't find the container with id 4198631295aced9fb4273be00a2162f9a0a741538a6941b964830afefba5434b
Sep 15 11:25:04.232143 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:04.232114    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc673a9ff7f802e0d691cff004f3253.slice/crio-72341310b1a777b3bd902b4afafb9301279e890ba7b117c714dac63aa14e466c WatchSource:0}: Error finding container 72341310b1a777b3bd902b4afafb9301279e890ba7b117c714dac63aa14e466c: Status 404 returned error can't find the container with id 72341310b1a777b3bd902b4afafb9301279e890ba7b117c714dac63aa14e466c
Sep 15 11:25:04.387949 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.387852    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs\") pod \"network-metrics-daemon-cc7mk\" (UID: \"355f2000-c95f-4d71-b450-03ce72b5ea62\") " pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:04.387949 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.387909    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret\") pod \"global-pull-secret-syncer-7j9vh\" (UID: \"85b6a362-7f90-4e54-89d2-2c21ef687efb\") " pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:04.388219 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:04.387997    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:04.388219 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:04.388011    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:04.388219 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.388026    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6gd\" (UniqueName: \"kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd\") pod \"network-check-target-4gbpc\" (UID: \"a0d6a68c-1fa6-4f31-b6f8-3a1a38988836\") " pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:04.388219 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:04.388084    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs podName:355f2000-c95f-4d71-b450-03ce72b5ea62 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:05.38804581 +0000 UTC m=+4.050540975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs") pod "network-metrics-daemon-cc7mk" (UID: "355f2000-c95f-4d71-b450-03ce72b5ea62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:04.388219 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:04.388144    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret podName:85b6a362-7f90-4e54-89d2-2c21ef687efb nodeName:}" failed. No retries permitted until 2026-09-15 11:25:05.388126666 +0000 UTC m=+4.050621826 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret") pod "global-pull-secret-syncer-7j9vh" (UID: "85b6a362-7f90-4e54-89d2-2c21ef687efb") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:04.388219 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:04.388163    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 11:25:04.388219 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:04.388187    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 11:25:04.388219 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:04.388202    2562 projected.go:196] Error preparing data for projected volume kube-api-access-6q6gd for pod openshift-network-diagnostics/network-check-target-4gbpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:04.388563 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:04.388262    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd podName:a0d6a68c-1fa6-4f31-b6f8-3a1a38988836 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:05.388246749 +0000 UTC m=+4.050741925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6q6gd" (UniqueName: "kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd") pod "network-check-target-4gbpc" (UID: "a0d6a68c-1fa6-4f31-b6f8-3a1a38988836") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:04.456230 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:04.456192    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515a11a1_5808_486b_b178_a3cf92c663d8.slice/crio-185356208f8a6c1d2fe2caf3aa7c45a053b994d2771c0551acd8439e761acec6 WatchSource:0}: Error finding container 185356208f8a6c1d2fe2caf3aa7c45a053b994d2771c0551acd8439e761acec6: Status 404 returned error can't find the container with id 185356208f8a6c1d2fe2caf3aa7c45a053b994d2771c0551acd8439e761acec6
Sep 15 11:25:04.562165 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:04.562130    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d28bab_39d1_4b87_bf7d_93dea93300f3.slice/crio-f32511a093780b9a5be51740fe1b05c9599c260d8e0a3533d94d3e2083d0ffce WatchSource:0}: Error finding container f32511a093780b9a5be51740fe1b05c9599c260d8e0a3533d94d3e2083d0ffce: Status 404 returned error can't find the container with id f32511a093780b9a5be51740fe1b05c9599c260d8e0a3533d94d3e2083d0ffce
Sep 15 11:25:04.801694 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.801589    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 11:20:03 +0000 UTC" deadline="2028-06-08 16:08:30.421191596 +0000 UTC"
Sep 15 11:25:04.801694 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.801633    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15172h43m25.619562915s"
Sep 15 11:25:04.868436 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.868403    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:04.868653 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:04.868569    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc7mk" podUID="355f2000-c95f-4d71-b450-03ce72b5ea62"
Sep 15 11:25:04.882403 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.882287    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qw6lz" event={"ID":"515a11a1-5808-486b-b178-a3cf92c663d8","Type":"ContainerStarted","Data":"185356208f8a6c1d2fe2caf3aa7c45a053b994d2771c0551acd8439e761acec6"}
Sep 15 11:25:04.898020 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.897934    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v8qdz" event={"ID":"c96758c8-f869-41dd-9350-e015e30b82df","Type":"ContainerStarted","Data":"4198631295aced9fb4273be00a2162f9a0a741538a6941b964830afefba5434b"}
Sep 15 11:25:04.911106 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.911066    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk259" event={"ID":"d662f7e1-efac-4ded-8c57-4aa36d0e84a0","Type":"ContainerStarted","Data":"54a9a1a9603f31e9ec179728ba9b91c8d5346345fd4af53e7f2808f44e08af78"}
Sep 15 11:25:04.938208 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.929986    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" event={"ID":"2a68a194c773c52a06b82b6c37030522","Type":"ContainerStarted","Data":"f7d050542c1542422f05c25cf24ec2b430c786081c2d859c461e861c18c40f11"}
Sep 15 11:25:04.956183 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.955938    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-42tpl" event={"ID":"85bb9ddc-8249-4923-a96d-4b17b5450c51","Type":"ContainerStarted","Data":"0390ea7c6f91b2842b64f21fa76c622dd3a2806b22b83d33ddf217ad4b3680af"}
Sep 15 11:25:04.960070 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.959122    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal" event={"ID":"4fc673a9ff7f802e0d691cff004f3253","Type":"ContainerStarted","Data":"72341310b1a777b3bd902b4afafb9301279e890ba7b117c714dac63aa14e466c"}
Sep 15 11:25:04.970493 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.970456    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sr9gc" event={"ID":"870be114-239c-4f62-8def-c3144e7cbeb0","Type":"ContainerStarted","Data":"eac4cbdc24d52a177bb5c35fce931451c61ecf488ed24e43c8b16ef3fa251baf"}
Sep 15 11:25:04.983249 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:04.983215    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e8a164_314f_4cfd_8121_edea9ed65041.slice/crio-f19a0c7b6e8b9c74dec9917861c374c3be1747f310e100d69f8281e6dd6af50a WatchSource:0}: Error finding container f19a0c7b6e8b9c74dec9917861c374c3be1747f310e100d69f8281e6dd6af50a: Status 404 returned error can't find the container with id f19a0c7b6e8b9c74dec9917861c374c3be1747f310e100d69f8281e6dd6af50a
Sep 15 11:25:04.985905 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.985872    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f5l6r" event={"ID":"dbb4a488-5113-44e5-bea0-b831896d7cac","Type":"ContainerStarted","Data":"65bf81bc3fd1353265ce2f279246156bc132c8a368981a62c4325cde69912de7"}
Sep 15 11:25:04.992701 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.992667    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl" event={"ID":"e311c2ba-68b7-47af-92c3-a8dac7d2d28b","Type":"ContainerStarted","Data":"4792228d3643dd04792310a8bb96b822d9c46392d4e90eee84a576767895b5e0"}
Sep 15 11:25:04.998278 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:04.998249    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8" event={"ID":"f5d28bab-39d1-4b87-bf7d-93dea93300f3","Type":"ContainerStarted","Data":"f32511a093780b9a5be51740fe1b05c9599c260d8e0a3533d94d3e2083d0ffce"}
Sep 15 11:25:05.401204 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:05.401144    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs\") pod \"network-metrics-daemon-cc7mk\" (UID: \"355f2000-c95f-4d71-b450-03ce72b5ea62\") " pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:05.401415 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:05.401232    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret\") pod \"global-pull-secret-syncer-7j9vh\" (UID: \"85b6a362-7f90-4e54-89d2-2c21ef687efb\") " pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:05.401415 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:05.401277    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6gd\" (UniqueName: \"kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd\") pod \"network-check-target-4gbpc\" (UID: \"a0d6a68c-1fa6-4f31-b6f8-3a1a38988836\") " pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:05.401556 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:05.401434    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 11:25:05.401556 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:05.401460    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 11:25:05.401556 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:05.401482    2562 projected.go:196] Error preparing data for projected volume kube-api-access-6q6gd for pod openshift-network-diagnostics/network-check-target-4gbpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:05.401556 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:05.401545    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd podName:a0d6a68c-1fa6-4f31-b6f8-3a1a38988836 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:07.401524994 +0000 UTC m=+6.064020175 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6q6gd" (UniqueName: "kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd") pod "network-check-target-4gbpc" (UID: "a0d6a68c-1fa6-4f31-b6f8-3a1a38988836") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:05.401984 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:05.401965    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:05.402062 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:05.402022    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs podName:355f2000-c95f-4d71-b450-03ce72b5ea62 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:07.402005213 +0000 UTC m=+6.064500377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs") pod "network-metrics-daemon-cc7mk" (UID: "355f2000-c95f-4d71-b450-03ce72b5ea62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:05.402163 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:05.402083    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:05.402163 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:05.402118    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret podName:85b6a362-7f90-4e54-89d2-2c21ef687efb nodeName:}" failed. No retries permitted until 2026-09-15 11:25:07.402106423 +0000 UTC m=+6.064601589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret") pod "global-pull-secret-syncer-7j9vh" (UID: "85b6a362-7f90-4e54-89d2-2c21ef687efb") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:05.802462 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:05.802332    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 11:20:03 +0000 UTC" deadline="2028-04-28 13:47:23.886948982 +0000 UTC"
Sep 15 11:25:05.802462 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:05.802405    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14186h22m18.08454879s"
Sep 15 11:25:05.870218 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:05.870180    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:05.870397 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:05.870328    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7j9vh" podUID="85b6a362-7f90-4e54-89d2-2c21ef687efb"
Sep 15 11:25:05.870767 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:05.870746    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:05.870875 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:05.870849    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gbpc" podUID="a0d6a68c-1fa6-4f31-b6f8-3a1a38988836"
Sep 15 11:25:06.028418 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:06.028376    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mfm26" event={"ID":"38e8a164-314f-4cfd-8121-edea9ed65041","Type":"ContainerStarted","Data":"f19a0c7b6e8b9c74dec9917861c374c3be1747f310e100d69f8281e6dd6af50a"}
Sep 15 11:25:06.867823 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:06.867735    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:06.868417 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:06.867879    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc7mk" podUID="355f2000-c95f-4d71-b450-03ce72b5ea62"
Sep 15 11:25:07.418144 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:07.418071    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret\") pod \"global-pull-secret-syncer-7j9vh\" (UID: \"85b6a362-7f90-4e54-89d2-2c21ef687efb\") " pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:07.418144 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:07.418141    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6gd\" (UniqueName: \"kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd\") pod \"network-check-target-4gbpc\" (UID: \"a0d6a68c-1fa6-4f31-b6f8-3a1a38988836\") " pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:07.418396 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:07.418196    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs\") pod \"network-metrics-daemon-cc7mk\" (UID: \"355f2000-c95f-4d71-b450-03ce72b5ea62\") " pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:07.418396 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:07.418336    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:07.418501 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:07.418399    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs podName:355f2000-c95f-4d71-b450-03ce72b5ea62 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:11.418380103 +0000 UTC m=+10.080875277 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs") pod "network-metrics-daemon-cc7mk" (UID: "355f2000-c95f-4d71-b450-03ce72b5ea62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:07.418840 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:07.418819    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:07.418921 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:07.418885    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret podName:85b6a362-7f90-4e54-89d2-2c21ef687efb nodeName:}" failed. No retries permitted until 2026-09-15 11:25:11.41886208 +0000 UTC m=+10.081357254 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret") pod "global-pull-secret-syncer-7j9vh" (UID: "85b6a362-7f90-4e54-89d2-2c21ef687efb") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:07.418991 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:07.418971    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 11:25:07.419042 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:07.418994    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 11:25:07.419042 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:07.419007    2562 projected.go:196] Error preparing data for projected volume kube-api-access-6q6gd for pod openshift-network-diagnostics/network-check-target-4gbpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:07.419165 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:07.419079    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd podName:a0d6a68c-1fa6-4f31-b6f8-3a1a38988836 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:11.419052189 +0000 UTC m=+10.081547348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6q6gd" (UniqueName: "kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd") pod "network-check-target-4gbpc" (UID: "a0d6a68c-1fa6-4f31-b6f8-3a1a38988836") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:07.869258 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:07.868979    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:07.869258 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:07.869138    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gbpc" podUID="a0d6a68c-1fa6-4f31-b6f8-3a1a38988836"
Sep 15 11:25:07.869783 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:07.869586    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:07.869783 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:07.869687    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7j9vh" podUID="85b6a362-7f90-4e54-89d2-2c21ef687efb"
Sep 15 11:25:08.868553 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:08.868197    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:08.868553 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:08.868350    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc7mk" podUID="355f2000-c95f-4d71-b450-03ce72b5ea62"
Sep 15 11:25:09.867590 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:09.867513    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:09.868082 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:09.867655    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7j9vh" podUID="85b6a362-7f90-4e54-89d2-2c21ef687efb"
Sep 15 11:25:09.868082 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:09.867721    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:09.868082 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:09.867799    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gbpc" podUID="a0d6a68c-1fa6-4f31-b6f8-3a1a38988836"
Sep 15 11:25:10.867928 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:10.867890    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:10.868421 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:10.868031    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc7mk" podUID="355f2000-c95f-4d71-b450-03ce72b5ea62"
Sep 15 11:25:11.454644 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:11.454118    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs\") pod \"network-metrics-daemon-cc7mk\" (UID: \"355f2000-c95f-4d71-b450-03ce72b5ea62\") " pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:11.454644 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:11.454184    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret\") pod \"global-pull-secret-syncer-7j9vh\" (UID: \"85b6a362-7f90-4e54-89d2-2c21ef687efb\") " pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:11.454644 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:11.454225    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6gd\" (UniqueName: \"kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd\") pod \"network-check-target-4gbpc\" (UID: \"a0d6a68c-1fa6-4f31-b6f8-3a1a38988836\") " pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:11.454644 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:11.454383    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 11:25:11.454644 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:11.454404    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 11:25:11.454644 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:11.454417    2562 projected.go:196] Error preparing data for projected volume kube-api-access-6q6gd for pod openshift-network-diagnostics/network-check-target-4gbpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:11.454644 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:11.454476    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd podName:a0d6a68c-1fa6-4f31-b6f8-3a1a38988836 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:19.454456708 +0000 UTC m=+18.116951887 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6q6gd" (UniqueName: "kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd") pod "network-check-target-4gbpc" (UID: "a0d6a68c-1fa6-4f31-b6f8-3a1a38988836") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:11.455123 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:11.454678    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:11.455123 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:11.454748    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret podName:85b6a362-7f90-4e54-89d2-2c21ef687efb nodeName:}" failed. No retries permitted until 2026-09-15 11:25:19.454729659 +0000 UTC m=+18.117224831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret") pod "global-pull-secret-syncer-7j9vh" (UID: "85b6a362-7f90-4e54-89d2-2c21ef687efb") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:11.455123 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:11.454687    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:11.455123 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:11.454803    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs podName:355f2000-c95f-4d71-b450-03ce72b5ea62 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:19.454795254 +0000 UTC m=+18.117290413 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs") pod "network-metrics-daemon-cc7mk" (UID: "355f2000-c95f-4d71-b450-03ce72b5ea62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:11.868734 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:11.868655    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:11.869132 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:11.868766    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gbpc" podUID="a0d6a68c-1fa6-4f31-b6f8-3a1a38988836"
Sep 15 11:25:11.869210 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:11.869189    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:11.869335 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:11.869308    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7j9vh" podUID="85b6a362-7f90-4e54-89d2-2c21ef687efb"
Sep 15 11:25:12.867524 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:12.867410    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:12.867755 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:12.867556    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc7mk" podUID="355f2000-c95f-4d71-b450-03ce72b5ea62"
Sep 15 11:25:13.867750 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:13.867651    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:13.867750 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:13.867699    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:13.868318 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:13.867800    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7j9vh" podUID="85b6a362-7f90-4e54-89d2-2c21ef687efb"
Sep 15 11:25:13.868318 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:13.867931    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gbpc" podUID="a0d6a68c-1fa6-4f31-b6f8-3a1a38988836"
Sep 15 11:25:14.868181 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:14.868149    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:14.868593 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:14.868281    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc7mk" podUID="355f2000-c95f-4d71-b450-03ce72b5ea62"
Sep 15 11:25:15.867354 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:15.867331    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:15.867354 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:15.867344    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:15.867506 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:15.867465    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7j9vh" podUID="85b6a362-7f90-4e54-89d2-2c21ef687efb"
Sep 15 11:25:15.867588 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:15.867563    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gbpc" podUID="a0d6a68c-1fa6-4f31-b6f8-3a1a38988836"
Sep 15 11:25:16.868232 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:16.868196    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:16.868669 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:16.868340    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc7mk" podUID="355f2000-c95f-4d71-b450-03ce72b5ea62"
Sep 15 11:25:17.867864 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:17.867827    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:17.868078 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:17.867971    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7j9vh" podUID="85b6a362-7f90-4e54-89d2-2c21ef687efb"
Sep 15 11:25:17.868078 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:17.868027    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:17.868203 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:17.868148    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gbpc" podUID="a0d6a68c-1fa6-4f31-b6f8-3a1a38988836"
Sep 15 11:25:18.868361 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:18.868320    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:18.868774 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:18.868452    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc7mk" podUID="355f2000-c95f-4d71-b450-03ce72b5ea62"
Sep 15 11:25:19.515014 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:19.514922    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret\") pod \"global-pull-secret-syncer-7j9vh\" (UID: \"85b6a362-7f90-4e54-89d2-2c21ef687efb\") " pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:19.515014 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:19.514983    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6gd\" (UniqueName: \"kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd\") pod \"network-check-target-4gbpc\" (UID: \"a0d6a68c-1fa6-4f31-b6f8-3a1a38988836\") " pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:19.515264 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:19.515034    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs\") pod \"network-metrics-daemon-cc7mk\" (UID: \"355f2000-c95f-4d71-b450-03ce72b5ea62\") " pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:19.515264 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:19.515109    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:19.515264 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:19.515136    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 11:25:19.515264 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:19.515155    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 11:25:19.515264 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:19.515157    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:19.515264 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:19.515168    2562 projected.go:196] Error preparing data for projected volume kube-api-access-6q6gd for pod openshift-network-diagnostics/network-check-target-4gbpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:19.515264 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:19.515187    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret podName:85b6a362-7f90-4e54-89d2-2c21ef687efb nodeName:}" failed. No retries permitted until 2026-09-15 11:25:35.515169334 +0000 UTC m=+34.177664512 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret") pod "global-pull-secret-syncer-7j9vh" (UID: "85b6a362-7f90-4e54-89d2-2c21ef687efb") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:19.515264 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:19.515206    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd podName:a0d6a68c-1fa6-4f31-b6f8-3a1a38988836 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:35.515195157 +0000 UTC m=+34.177690321 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6q6gd" (UniqueName: "kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd") pod "network-check-target-4gbpc" (UID: "a0d6a68c-1fa6-4f31-b6f8-3a1a38988836") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:19.515264 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:19.515224    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs podName:355f2000-c95f-4d71-b450-03ce72b5ea62 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:35.515215579 +0000 UTC m=+34.177710739 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs") pod "network-metrics-daemon-cc7mk" (UID: "355f2000-c95f-4d71-b450-03ce72b5ea62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:19.867927 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:19.867889    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:19.868121 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:19.868030    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7j9vh" podUID="85b6a362-7f90-4e54-89d2-2c21ef687efb"
Sep 15 11:25:19.868121 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:19.868089    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:19.868237 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:19.868199    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gbpc" podUID="a0d6a68c-1fa6-4f31-b6f8-3a1a38988836"
Sep 15 11:25:20.867854 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:20.867823    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:20.868336 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:20.867961    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc7mk" podUID="355f2000-c95f-4d71-b450-03ce72b5ea62"
Sep 15 11:25:21.868492 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:21.868453    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:21.868929 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:21.868553    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gbpc" podUID="a0d6a68c-1fa6-4f31-b6f8-3a1a38988836"
Sep 15 11:25:21.868929 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:21.868574    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:21.868929 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:21.868670    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7j9vh" podUID="85b6a362-7f90-4e54-89d2-2c21ef687efb"
Sep 15 11:25:22.867771 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:22.867733    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:22.867952 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:22.867869    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc7mk" podUID="355f2000-c95f-4d71-b450-03ce72b5ea62"
Sep 15 11:25:23.871787 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:23.871600    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:23.872248 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:23.871606    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:23.872248 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:23.871892    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7j9vh" podUID="85b6a362-7f90-4e54-89d2-2c21ef687efb"
Sep 15 11:25:23.872248 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:23.871924    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gbpc" podUID="a0d6a68c-1fa6-4f31-b6f8-3a1a38988836"
Sep 15 11:25:24.068946 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:24.068911    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v8qdz" event={"ID":"c96758c8-f869-41dd-9350-e015e30b82df","Type":"ContainerStarted","Data":"ce887152602860e8ae3baf6ea0e984b639f8caf523514721f4f305d316edc2a8"}
Sep 15 11:25:24.070496 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:24.070472    2562 generic.go:356] "Generic (PLEG): container finished" podID="d662f7e1-efac-4ded-8c57-4aa36d0e84a0" containerID="e9e594f46a564b2252ceaae3baa85ddfd886ceb79c829bb4d7aee5d3a3d93445" exitCode=0
Sep 15 11:25:24.070590 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:24.070531    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk259" event={"ID":"d662f7e1-efac-4ded-8c57-4aa36d0e84a0","Type":"ContainerDied","Data":"e9e594f46a564b2252ceaae3baa85ddfd886ceb79c829bb4d7aee5d3a3d93445"}
Sep 15 11:25:24.072110 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:24.072084    2562 generic.go:356] "Generic (PLEG): container finished" podID="2a68a194c773c52a06b82b6c37030522" containerID="ee2a8864f29308ca30757cbce16d2398978dfd5e9c4d354a4352fe9468a406e9" exitCode=0
Sep 15 11:25:24.072194 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:24.072165    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" event={"ID":"2a68a194c773c52a06b82b6c37030522","Type":"ContainerDied","Data":"ee2a8864f29308ca30757cbce16d2398978dfd5e9c4d354a4352fe9468a406e9"}
Sep 15 11:25:24.073791 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:24.073751    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-42tpl" event={"ID":"85bb9ddc-8249-4923-a96d-4b17b5450c51","Type":"ContainerStarted","Data":"d364aef9126219c0c39f338b71f977ec87c3f569ea2034731a8282b99e64cd4b"}
Sep 15 11:25:24.075688 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:24.075661    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mfm26" event={"ID":"38e8a164-314f-4cfd-8121-edea9ed65041","Type":"ContainerStarted","Data":"3dda2d3910085b429fea0dc02e0abea29dd157c5031b6056b72360a82f248513"}
Sep 15 11:25:24.077617 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:24.077586    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal" event={"ID":"4fc673a9ff7f802e0d691cff004f3253","Type":"ContainerStarted","Data":"bfdc81dff902b79520a9677e6886ebb4b30524cff14501ed11c1e861200a01a8"}
Sep 15 11:25:24.079335 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:24.079285    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sr9gc" event={"ID":"870be114-239c-4f62-8def-c3144e7cbeb0","Type":"ContainerStarted","Data":"21a75a2b5513e2c5a5ce78fa682328330d1b729f497d4df4f24bf3873b2c1806"}
Sep 15 11:25:24.080695 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:24.080667    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f5l6r" event={"ID":"dbb4a488-5113-44e5-bea0-b831896d7cac","Type":"ContainerStarted","Data":"ef31aba672f81a9c366090c5aecaf1c919721a23e43a7942b31c0583b36d3b23"}
Sep 15 11:25:24.082757 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:24.082738    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl" event={"ID":"e311c2ba-68b7-47af-92c3-a8dac7d2d28b","Type":"ContainerStarted","Data":"13e5200ab00cefdb9676af64ec0fe47202d394fa2affed2afcf4ecdee2a21493"}
Sep 15 11:25:24.082855 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:24.082835    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl" event={"ID":"e311c2ba-68b7-47af-92c3-a8dac7d2d28b","Type":"ContainerStarted","Data":"44ecc7fccbdd9aca61b3a0f0eedd44bdeffec549bd8ecea5a06708d7fc5f2503"}
Sep 15 11:25:24.084246 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:24.084223    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8" event={"ID":"f5d28bab-39d1-4b87-bf7d-93dea93300f3","Type":"ContainerStarted","Data":"1e080395669bbe01bdb6d41f963a15770e3030034e6de87f48ecd76fb14fe5fa"}
Sep 15 11:25:24.085395 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:24.085358    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-v8qdz" podStartSLOduration=2.810888859 podStartE2EDuration="22.08534602s" totalImagesPullingTime="19.274457161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:24.084947911 +0000 UTC m=+22.747443105" watchObservedRunningTime="2026-09-15 11:25:24.08534602 +0000 UTC m=+22.747841201"
Sep 15 11:25:24.098484 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:24.098445    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal" podStartSLOduration=21.098434113 podStartE2EDuration="21.098434113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:24.098359059 +0000 UTC m=+22.760854242" watchObservedRunningTime="2026-09-15 11:25:24.098434113 +0000 UTC m=+22.760929293"
Sep 15 11:25:24.114482 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:24.114443    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mfm26" podStartSLOduration=3.629346664 podStartE2EDuration="22.11442802s" totalImagesPullingTime="18.485081356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:24.114303173 +0000 UTC m=+22.776798355" watchObservedRunningTime="2026-09-15 11:25:24.11442802 +0000 UTC m=+22.776923246"
Sep 15 11:25:24.126963 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:24.126922    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-42tpl" podStartSLOduration=2.867013697 podStartE2EDuration="22.126904894s" totalImagesPullingTime="19.259891197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:24.126290628 +0000 UTC m=+22.788785809" watchObservedRunningTime="2026-09-15 11:25:24.126904894 +0000 UTC m=+22.789400076"
Sep 15 11:25:24.179637 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:24.179595    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-f5l6r" podStartSLOduration=2.89501115 podStartE2EDuration="22.179579339s" totalImagesPullingTime="19.284568189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:24.179243363 +0000 UTC m=+22.841738558" watchObservedRunningTime="2026-09-15 11:25:24.179579339 +0000 UTC m=+22.842074520"
Sep 15 11:25:24.868260 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:24.868051    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:24.868442 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:24.868352    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc7mk" podUID="355f2000-c95f-4d71-b450-03ce72b5ea62"
Sep 15 11:25:25.067914 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:25.067890    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 11:25:25.087931 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:25.087905    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" event={"ID":"2a68a194c773c52a06b82b6c37030522","Type":"ContainerStarted","Data":"00a82c119938541433aab6340e380545d989db83638a0873350d338553b0d287"}
Sep 15 11:25:25.091064 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:25.091033    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl" event={"ID":"e311c2ba-68b7-47af-92c3-a8dac7d2d28b","Type":"ContainerStarted","Data":"94f53c827f0d7466136c605d899f20df3c8a3633ad63c71aca9d5b2e03c37aee"}
Sep 15 11:25:25.091169 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:25.091110    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl" event={"ID":"e311c2ba-68b7-47af-92c3-a8dac7d2d28b","Type":"ContainerStarted","Data":"abc56d22ec5b399e3c106df4fcd03e9a0ac725426da429cf3b45598ec3c5a3b0"}
Sep 15 11:25:25.091169 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:25.091143    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl" event={"ID":"e311c2ba-68b7-47af-92c3-a8dac7d2d28b","Type":"ContainerStarted","Data":"2403b5c77615b5cc9326fe3bd84baade165c85ce9b0fcd6d0685606b0d1ca888"}
Sep 15 11:25:25.091250 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:25.091178    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl" event={"ID":"e311c2ba-68b7-47af-92c3-a8dac7d2d28b","Type":"ContainerStarted","Data":"6b7fc4716d468cdfed0c2a1fe9984d56687427fc6c59d8e1a017aec82887231a"}
Sep 15 11:25:25.092743 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:25.092721    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8" event={"ID":"f5d28bab-39d1-4b87-bf7d-93dea93300f3","Type":"ContainerStarted","Data":"0a0453b82dc05debc723615920384bd115edcff44819c87194a02dc80f0384ef"}
Sep 15 11:25:25.094096 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:25.094078    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qw6lz" event={"ID":"515a11a1-5808-486b-b178-a3cf92c663d8","Type":"ContainerStarted","Data":"d30fa65a468361702bb94bfdd3479ebc39df3ddbff81882a7b92dff318ea4664"}
Sep 15 11:25:25.101395 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:25.101355    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sr9gc" podStartSLOduration=11.494678305 podStartE2EDuration="23.101341773s" totalImagesPullingTime="11.606663468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:24.19304751 +0000 UTC m=+22.855542691" watchObservedRunningTime="2026-09-15 11:25:25.101341773 +0000 UTC m=+23.763836953"
Sep 15 11:25:25.101596 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:25.101514    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" podStartSLOduration=22.101505357 podStartE2EDuration="22.101505357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:25.101197317 +0000 UTC m=+23.763692498" watchObservedRunningTime="2026-09-15 11:25:25.101505357 +0000 UTC m=+23.764000540"
Sep 15 11:25:25.577436 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:25.577394    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sr9gc"
Sep 15 11:25:25.577974 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:25.577952    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sr9gc"
Sep 15 11:25:25.592219 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:25.592180    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qw6lz" podStartSLOduration=4.607160314 podStartE2EDuration="23.592164979s" totalImagesPullingTime="18.985004665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:25.114328837 +0000 UTC m=+23.776824018" watchObservedRunningTime="2026-09-15 11:25:25.592164979 +0000 UTC m=+24.254660192"
Sep 15 11:25:25.837678 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:25.837567    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T11:25:25.067908201Z","UUID":"13af1710-a427-4305-a0e3-6c667b0c41e5","Handler":null,"Name":"","Endpoint":""}
Sep 15 11:25:25.839302 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:25.839276    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 11:25:25.839449 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:25.839310    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 11:25:25.867540 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:25.867513    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:25.867540 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:25.867527    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:25.867777 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:25.867637    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gbpc" podUID="a0d6a68c-1fa6-4f31-b6f8-3a1a38988836"
Sep 15 11:25:25.867777 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:25.867757    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7j9vh" podUID="85b6a362-7f90-4e54-89d2-2c21ef687efb"
Sep 15 11:25:26.868117 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:26.868085    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:26.868521 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:26.868229    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc7mk" podUID="355f2000-c95f-4d71-b450-03ce72b5ea62"
Sep 15 11:25:27.103948 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:27.103898    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl" event={"ID":"e311c2ba-68b7-47af-92c3-a8dac7d2d28b","Type":"ContainerStarted","Data":"e309291b3b4883137f08a78a16ecec55e96f467a247d2c803f86796e9bef1153"}
Sep 15 11:25:27.104128 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:27.103926    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 15 11:25:27.867514 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:27.867476    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:27.867514 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:27.867512    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:27.867762 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:27.867609    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7j9vh" podUID="85b6a362-7f90-4e54-89d2-2c21ef687efb"
Sep 15 11:25:27.867762 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:27.867720    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gbpc" podUID="a0d6a68c-1fa6-4f31-b6f8-3a1a38988836"
Sep 15 11:25:28.110016 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:28.109978    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8" event={"ID":"f5d28bab-39d1-4b87-bf7d-93dea93300f3","Type":"ContainerStarted","Data":"3e6c36b27b2a9cbebac2be17684e4a4b5f8d5c6cda47b7d3e9f8e930930e0dd0"}
Sep 15 11:25:28.127813 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:28.127716    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jcff8" podStartSLOduration=3.971465116 podStartE2EDuration="26.127700427s" totalImagesPullingTime="22.156235311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:28.127211698 +0000 UTC m=+26.789706907" watchObservedRunningTime="2026-09-15 11:25:28.127700427 +0000 UTC m=+26.790195608"
Sep 15 11:25:28.867970 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:28.867930    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:28.868156 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:28.868077    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc7mk" podUID="355f2000-c95f-4d71-b450-03ce72b5ea62"
Sep 15 11:25:29.867800 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:29.867608    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:29.868420 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:29.867614    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:29.868420 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:29.867892    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7j9vh" podUID="85b6a362-7f90-4e54-89d2-2c21ef687efb"
Sep 15 11:25:29.868420 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:29.867945    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gbpc" podUID="a0d6a68c-1fa6-4f31-b6f8-3a1a38988836"
Sep 15 11:25:30.116400 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:30.116365    2562 generic.go:356] "Generic (PLEG): container finished" podID="d662f7e1-efac-4ded-8c57-4aa36d0e84a0" containerID="c79e3091433855073f80fe9ae23019038feae014dd81c8e73cff8b71e1c7619e" exitCode=0
Sep 15 11:25:30.116573 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:30.116450    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk259" event={"ID":"d662f7e1-efac-4ded-8c57-4aa36d0e84a0","Type":"ContainerDied","Data":"c79e3091433855073f80fe9ae23019038feae014dd81c8e73cff8b71e1c7619e"}
Sep 15 11:25:30.120531 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:30.120507    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl" event={"ID":"e311c2ba-68b7-47af-92c3-a8dac7d2d28b","Type":"ContainerStarted","Data":"af5f7f79b271b2a95941a018987773de93e3aa092befed293ce8791f65a2f81a"}
Sep 15 11:25:30.120839 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:30.120811    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:30.135576 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:30.135555    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:30.172440 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:30.172388    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl" podStartSLOduration=8.67928499 podStartE2EDuration="28.172372324s" totalImagesPullingTime="19.493087334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:30.171491623 +0000 UTC m=+28.833986818" watchObservedRunningTime="2026-09-15 11:25:30.172372324 +0000 UTC m=+28.834867505"
Sep 15 11:25:30.867979 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:30.867942    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:30.868682 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:30.868120    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc7mk" podUID="355f2000-c95f-4d71-b450-03ce72b5ea62"
Sep 15 11:25:31.124810 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:31.124622    2562 generic.go:356] "Generic (PLEG): container finished" podID="d662f7e1-efac-4ded-8c57-4aa36d0e84a0" containerID="32c825c9b303eb20b19e0f020483eacb25217f1bed352aebe34f2c3504fa0cf5" exitCode=0
Sep 15 11:25:31.125032 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:31.124712    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk259" event={"ID":"d662f7e1-efac-4ded-8c57-4aa36d0e84a0","Type":"ContainerDied","Data":"32c825c9b303eb20b19e0f020483eacb25217f1bed352aebe34f2c3504fa0cf5"}
Sep 15 11:25:31.125339 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:31.125318    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:31.125431 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:31.125381    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:31.140698 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:31.140676    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:25:31.206619 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:31.206585    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4gbpc"]
Sep 15 11:25:31.206807 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:31.206722    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:31.206873 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:31.206811    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gbpc" podUID="a0d6a68c-1fa6-4f31-b6f8-3a1a38988836"
Sep 15 11:25:31.209654 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:31.209628    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7j9vh"]
Sep 15 11:25:31.209747 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:31.209714    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:31.209800 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:31.209785    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7j9vh" podUID="85b6a362-7f90-4e54-89d2-2c21ef687efb"
Sep 15 11:25:31.212751 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:31.212732    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cc7mk"]
Sep 15 11:25:31.212829 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:31.212814    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:31.212911 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:31.212893    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc7mk" podUID="355f2000-c95f-4d71-b450-03ce72b5ea62"
Sep 15 11:25:31.451186 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:31.451100    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sr9gc"
Sep 15 11:25:31.451320 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:31.451234    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 15 11:25:31.451879 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:31.451848    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sr9gc"
Sep 15 11:25:32.867896 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:32.867847    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:32.867896 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:32.867889    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:32.868415 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:32.867889    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:32.868415 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:32.868004    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7j9vh" podUID="85b6a362-7f90-4e54-89d2-2c21ef687efb"
Sep 15 11:25:32.868415 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:32.868168    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc7mk" podUID="355f2000-c95f-4d71-b450-03ce72b5ea62"
Sep 15 11:25:32.868415 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:32.868239    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gbpc" podUID="a0d6a68c-1fa6-4f31-b6f8-3a1a38988836"
Sep 15 11:25:34.134711 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:34.134676    2562 generic.go:356] "Generic (PLEG): container finished" podID="d662f7e1-efac-4ded-8c57-4aa36d0e84a0" containerID="6c1df56cd61dd13e697a9f430a41ed6d326bdec1e9691615d3cf8806c7530f3c" exitCode=0
Sep 15 11:25:34.135103 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:34.134734    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk259" event={"ID":"d662f7e1-efac-4ded-8c57-4aa36d0e84a0","Type":"ContainerDied","Data":"6c1df56cd61dd13e697a9f430a41ed6d326bdec1e9691615d3cf8806c7530f3c"}
Sep 15 11:25:34.868233 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:34.868199    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:34.868233 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:34.868220    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:34.868233 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:34.868224    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:34.868478 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:34.868350    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc7mk" podUID="355f2000-c95f-4d71-b450-03ce72b5ea62"
Sep 15 11:25:34.868530 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:34.868481    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gbpc" podUID="a0d6a68c-1fa6-4f31-b6f8-3a1a38988836"
Sep 15 11:25:34.868599 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:34.868571    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7j9vh" podUID="85b6a362-7f90-4e54-89d2-2c21ef687efb"
Sep 15 11:25:35.523039 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:35.523005    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret\") pod \"global-pull-secret-syncer-7j9vh\" (UID: \"85b6a362-7f90-4e54-89d2-2c21ef687efb\") " pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:35.523496 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:35.523125    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6gd\" (UniqueName: \"kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd\") pod \"network-check-target-4gbpc\" (UID: \"a0d6a68c-1fa6-4f31-b6f8-3a1a38988836\") " pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:35.523496 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:35.523192    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:35.523496 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:35.523212    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs\") pod \"network-metrics-daemon-cc7mk\" (UID: \"355f2000-c95f-4d71-b450-03ce72b5ea62\") " pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:35.523496 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:35.523263    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret podName:85b6a362-7f90-4e54-89d2-2c21ef687efb nodeName:}" failed. No retries permitted until 2026-09-15 11:26:07.523242605 +0000 UTC m=+66.185737763 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret") pod "global-pull-secret-syncer-7j9vh" (UID: "85b6a362-7f90-4e54-89d2-2c21ef687efb") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:35.523496 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:35.523275    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 11:25:35.523496 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:35.523299    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 11:25:35.523496 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:35.523312    2562 projected.go:196] Error preparing data for projected volume kube-api-access-6q6gd for pod openshift-network-diagnostics/network-check-target-4gbpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:35.523496 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:35.523324    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:35.523496 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:35.523374    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd podName:a0d6a68c-1fa6-4f31-b6f8-3a1a38988836 nodeName:}" failed. No retries permitted until 2026-09-15 11:26:07.523349242 +0000 UTC m=+66.185844417 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6q6gd" (UniqueName: "kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd") pod "network-check-target-4gbpc" (UID: "a0d6a68c-1fa6-4f31-b6f8-3a1a38988836") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:35.523496 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:35.523397    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs podName:355f2000-c95f-4d71-b450-03ce72b5ea62 nodeName:}" failed. No retries permitted until 2026-09-15 11:26:07.523386801 +0000 UTC m=+66.185881964 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs") pod "network-metrics-daemon-cc7mk" (UID: "355f2000-c95f-4d71-b450-03ce72b5ea62") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:36.868224 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:36.868188    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:36.868649 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:36.868233    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:36.868649 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:36.868342    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:36.868649 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:36.868342    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gbpc" podUID="a0d6a68c-1fa6-4f31-b6f8-3a1a38988836"
Sep 15 11:25:36.868649 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:36.868435    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7j9vh" podUID="85b6a362-7f90-4e54-89d2-2c21ef687efb"
Sep 15 11:25:36.868649 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:36.868512    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cc7mk" podUID="355f2000-c95f-4d71-b450-03ce72b5ea62"
Sep 15 11:25:37.656791 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.656537    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeReady"
Sep 15 11:25:37.657010 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.656919    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 11:25:37.704673 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.704633    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9wtp8"]
Sep 15 11:25:37.731457 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.731421    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tgsbr"]
Sep 15 11:25:37.731628 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.731574    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:25:37.733634 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.733602    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 11:25:37.733788 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.733768    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 11:25:37.733926 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.733908    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-55wz6\""
Sep 15 11:25:37.745270 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.745245    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9wtp8"]
Sep 15 11:25:37.745422 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.745288    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tgsbr"]
Sep 15 11:25:37.745422 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.745380    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tgsbr"
Sep 15 11:25:37.747805 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.747770    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 11:25:37.747959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.747813    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 11:25:37.747959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.747919    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fh4vk\""
Sep 15 11:25:37.748197 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.748181    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 11:25:37.842585 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.842546    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b887a34-bd19-4fa3-9897-a03605d5231c-config-volume\") pod \"dns-default-9wtp8\" (UID: \"8b887a34-bd19-4fa3-9897-a03605d5231c\") " pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:25:37.842585 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.842591    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkl6r\" (UniqueName: \"kubernetes.io/projected/8b887a34-bd19-4fa3-9897-a03605d5231c-kube-api-access-gkl6r\") pod \"dns-default-9wtp8\" (UID: \"8b887a34-bd19-4fa3-9897-a03605d5231c\") " pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:25:37.842837 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.842663    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9jfc\" (UniqueName: \"kubernetes.io/projected/356a69de-a242-4342-87a5-f876c681dd1e-kube-api-access-k9jfc\") pod \"ingress-canary-tgsbr\" (UID: \"356a69de-a242-4342-87a5-f876c681dd1e\") " pod="openshift-ingress-canary/ingress-canary-tgsbr"
Sep 15 11:25:37.842837 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.842687    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/356a69de-a242-4342-87a5-f876c681dd1e-cert\") pod \"ingress-canary-tgsbr\" (UID: \"356a69de-a242-4342-87a5-f876c681dd1e\") " pod="openshift-ingress-canary/ingress-canary-tgsbr"
Sep 15 11:25:37.842837 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.842721    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b887a34-bd19-4fa3-9897-a03605d5231c-metrics-tls\") pod \"dns-default-9wtp8\" (UID: \"8b887a34-bd19-4fa3-9897-a03605d5231c\") " pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:25:37.842837 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.842768    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b887a34-bd19-4fa3-9897-a03605d5231c-tmp-dir\") pod \"dns-default-9wtp8\" (UID: \"8b887a34-bd19-4fa3-9897-a03605d5231c\") " pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:25:37.943765 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.943672    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b887a34-bd19-4fa3-9897-a03605d5231c-config-volume\") pod \"dns-default-9wtp8\" (UID: \"8b887a34-bd19-4fa3-9897-a03605d5231c\") " pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:25:37.943765 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.943728    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkl6r\" (UniqueName: \"kubernetes.io/projected/8b887a34-bd19-4fa3-9897-a03605d5231c-kube-api-access-gkl6r\") pod \"dns-default-9wtp8\" (UID: \"8b887a34-bd19-4fa3-9897-a03605d5231c\") " pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:25:37.944286 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.944016    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9jfc\" (UniqueName: \"kubernetes.io/projected/356a69de-a242-4342-87a5-f876c681dd1e-kube-api-access-k9jfc\") pod \"ingress-canary-tgsbr\" (UID: \"356a69de-a242-4342-87a5-f876c681dd1e\") " pod="openshift-ingress-canary/ingress-canary-tgsbr"
Sep 15 11:25:37.944286 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.944074    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/356a69de-a242-4342-87a5-f876c681dd1e-cert\") pod \"ingress-canary-tgsbr\" (UID: \"356a69de-a242-4342-87a5-f876c681dd1e\") " pod="openshift-ingress-canary/ingress-canary-tgsbr"
Sep 15 11:25:37.944286 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.944114    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b887a34-bd19-4fa3-9897-a03605d5231c-metrics-tls\") pod \"dns-default-9wtp8\" (UID: \"8b887a34-bd19-4fa3-9897-a03605d5231c\") " pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:25:37.944286 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.944168    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b887a34-bd19-4fa3-9897-a03605d5231c-tmp-dir\") pod \"dns-default-9wtp8\" (UID: \"8b887a34-bd19-4fa3-9897-a03605d5231c\") " pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:25:37.944481 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:37.944296    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 11:25:37.944481 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:37.944361    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b887a34-bd19-4fa3-9897-a03605d5231c-metrics-tls podName:8b887a34-bd19-4fa3-9897-a03605d5231c nodeName:}" failed. No retries permitted until 2026-09-15 11:25:38.444340632 +0000 UTC m=+37.106835807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b887a34-bd19-4fa3-9897-a03605d5231c-metrics-tls") pod "dns-default-9wtp8" (UID: "8b887a34-bd19-4fa3-9897-a03605d5231c") : secret "dns-default-metrics-tls" not found
Sep 15 11:25:37.944571 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.944485    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b887a34-bd19-4fa3-9897-a03605d5231c-tmp-dir\") pod \"dns-default-9wtp8\" (UID: \"8b887a34-bd19-4fa3-9897-a03605d5231c\") " pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:25:37.944680 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.944651    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b887a34-bd19-4fa3-9897-a03605d5231c-config-volume\") pod \"dns-default-9wtp8\" (UID: \"8b887a34-bd19-4fa3-9897-a03605d5231c\") " pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:25:37.948850 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.948815    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/356a69de-a242-4342-87a5-f876c681dd1e-cert\") pod \"ingress-canary-tgsbr\" (UID: \"356a69de-a242-4342-87a5-f876c681dd1e\") " pod="openshift-ingress-canary/ingress-canary-tgsbr"
Sep 15 11:25:37.952996 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.952967    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9jfc\" (UniqueName: \"kubernetes.io/projected/356a69de-a242-4342-87a5-f876c681dd1e-kube-api-access-k9jfc\") pod \"ingress-canary-tgsbr\" (UID: \"356a69de-a242-4342-87a5-f876c681dd1e\") " pod="openshift-ingress-canary/ingress-canary-tgsbr"
Sep 15 11:25:37.953202 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:37.953180    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkl6r\" (UniqueName: \"kubernetes.io/projected/8b887a34-bd19-4fa3-9897-a03605d5231c-kube-api-access-gkl6r\") pod \"dns-default-9wtp8\" (UID: \"8b887a34-bd19-4fa3-9897-a03605d5231c\") " pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:25:38.055788 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:38.055748    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tgsbr"
Sep 15 11:25:38.448117 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:38.448090    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b887a34-bd19-4fa3-9897-a03605d5231c-metrics-tls\") pod \"dns-default-9wtp8\" (UID: \"8b887a34-bd19-4fa3-9897-a03605d5231c\") " pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:25:38.448281 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:38.448265    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 11:25:38.448347 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:38.448335    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b887a34-bd19-4fa3-9897-a03605d5231c-metrics-tls podName:8b887a34-bd19-4fa3-9897-a03605d5231c nodeName:}" failed. No retries permitted until 2026-09-15 11:25:39.448314914 +0000 UTC m=+38.110810074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b887a34-bd19-4fa3-9897-a03605d5231c-metrics-tls") pod "dns-default-9wtp8" (UID: "8b887a34-bd19-4fa3-9897-a03605d5231c") : secret "dns-default-metrics-tls" not found
Sep 15 11:25:38.517553 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:38.517523    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tgsbr"]
Sep 15 11:25:38.522916 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:38.522890    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356a69de_a242_4342_87a5_f876c681dd1e.slice/crio-87d2df5c6ed344102d2e301cbbf34df338dc3e63a29df0ac12ec8370a5cf9135 WatchSource:0}: Error finding container 87d2df5c6ed344102d2e301cbbf34df338dc3e63a29df0ac12ec8370a5cf9135: Status 404 returned error can't find the container with id 87d2df5c6ed344102d2e301cbbf34df338dc3e63a29df0ac12ec8370a5cf9135
Sep 15 11:25:38.867405 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:38.867373    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:25:38.867571 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:38.867374    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:25:38.867606 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:38.867378    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:25:38.869778 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:38.869757    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 11:25:38.870361 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:38.870347    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 11:25:38.870427 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:38.870382    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n59hg\""
Sep 15 11:25:38.870690 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:38.870677    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 11:25:38.870690 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:38.870683    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4fvsz\""
Sep 15 11:25:38.870828 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:38.870695    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 11:25:39.148427 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:39.148343    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tgsbr" event={"ID":"356a69de-a242-4342-87a5-f876c681dd1e","Type":"ContainerStarted","Data":"87d2df5c6ed344102d2e301cbbf34df338dc3e63a29df0ac12ec8370a5cf9135"}
Sep 15 11:25:39.151260 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:39.151042    2562 generic.go:356] "Generic (PLEG): container finished" podID="d662f7e1-efac-4ded-8c57-4aa36d0e84a0" containerID="7c01adc21fd8265f93ef40582caf63383bce3550cd73ad9b05255a6c447f8cb7" exitCode=0
Sep 15 11:25:39.151391 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:39.151129    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk259" event={"ID":"d662f7e1-efac-4ded-8c57-4aa36d0e84a0","Type":"ContainerDied","Data":"7c01adc21fd8265f93ef40582caf63383bce3550cd73ad9b05255a6c447f8cb7"}
Sep 15 11:25:39.456039 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:39.455923    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b887a34-bd19-4fa3-9897-a03605d5231c-metrics-tls\") pod \"dns-default-9wtp8\" (UID: \"8b887a34-bd19-4fa3-9897-a03605d5231c\") " pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:25:39.456200 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:39.456091    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 11:25:39.456200 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:39.456172    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b887a34-bd19-4fa3-9897-a03605d5231c-metrics-tls podName:8b887a34-bd19-4fa3-9897-a03605d5231c nodeName:}" failed. No retries permitted until 2026-09-15 11:25:41.456152967 +0000 UTC m=+40.118648126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b887a34-bd19-4fa3-9897-a03605d5231c-metrics-tls") pod "dns-default-9wtp8" (UID: "8b887a34-bd19-4fa3-9897-a03605d5231c") : secret "dns-default-metrics-tls" not found
Sep 15 11:25:40.157290 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:40.157255    2562 generic.go:356] "Generic (PLEG): container finished" podID="d662f7e1-efac-4ded-8c57-4aa36d0e84a0" containerID="53a42bf111169e0dd42c7b83166424458c2364694864aadb4a0dae6eeee59185" exitCode=0
Sep 15 11:25:40.157792 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:40.157316    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk259" event={"ID":"d662f7e1-efac-4ded-8c57-4aa36d0e84a0","Type":"ContainerDied","Data":"53a42bf111169e0dd42c7b83166424458c2364694864aadb4a0dae6eeee59185"}
Sep 15 11:25:41.164240 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.164209    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk259" event={"ID":"d662f7e1-efac-4ded-8c57-4aa36d0e84a0","Type":"ContainerStarted","Data":"f4f0a8a09ffe69e5125031f27cc302a20be6478ecc45f54666b00dc030563bc8"}
Sep 15 11:25:41.188351 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.188297    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qk259" podStartSLOduration=7.343738452 podStartE2EDuration="39.188274695s" totalImagesPullingTime="31.844536243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:41.186540756 +0000 UTC m=+39.849035960" watchObservedRunningTime="2026-09-15 11:25:41.188274695 +0000 UTC m=+39.850769877"
Sep 15 11:25:41.264362 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.264328    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rs74z"]
Sep 15 11:25:41.281796 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.281769    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:41.285458 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.285428    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 11:25:41.285595 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.285471    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8vhk8\""
Sep 15 11:25:41.285595 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.285419    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 11:25:41.285704 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.285617    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 11:25:41.288220 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.288073    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 11:25:41.295962 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.295932    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rs74z"]
Sep 15 11:25:41.369687 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.369647    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/27c60b28-4026-4737-a4f2-b69c08522dae-data-volume\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:41.369859 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.369723    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27c60b28-4026-4737-a4f2-b69c08522dae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:41.369859 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.369777    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/27c60b28-4026-4737-a4f2-b69c08522dae-crio-socket\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:41.369859 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.369814    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27c60b28-4026-4737-a4f2-b69c08522dae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:41.369985 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.369877    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqg5w\" (UniqueName: \"kubernetes.io/projected/27c60b28-4026-4737-a4f2-b69c08522dae-kube-api-access-lqg5w\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:41.471149 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.471051    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27c60b28-4026-4737-a4f2-b69c08522dae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:41.471149 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.471109    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/27c60b28-4026-4737-a4f2-b69c08522dae-crio-socket\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:41.471149 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.471138    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27c60b28-4026-4737-a4f2-b69c08522dae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:41.471426 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:41.471245    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 11:25:41.471426 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:41.471277    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 11:25:41.471426 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.471306    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqg5w\" (UniqueName: \"kubernetes.io/projected/27c60b28-4026-4737-a4f2-b69c08522dae-kube-api-access-lqg5w\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:41.471426 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:41.471348    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27c60b28-4026-4737-a4f2-b69c08522dae-kube-rbac-proxy-cm podName:27c60b28-4026-4737-a4f2-b69c08522dae nodeName:}" failed. No retries permitted until 2026-09-15 11:25:41.971327106 +0000 UTC m=+40.633822267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/27c60b28-4026-4737-a4f2-b69c08522dae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rs74z" (UID: "27c60b28-4026-4737-a4f2-b69c08522dae") : configmap "kube-rbac-proxy" not found
Sep 15 11:25:41.471426 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.471414    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/27c60b28-4026-4737-a4f2-b69c08522dae-crio-socket\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:41.471674 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:41.471431    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27c60b28-4026-4737-a4f2-b69c08522dae-insights-runtime-extractor-tls podName:27c60b28-4026-4737-a4f2-b69c08522dae nodeName:}" failed. No retries permitted until 2026-09-15 11:25:41.971393392 +0000 UTC m=+40.633892306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27c60b28-4026-4737-a4f2-b69c08522dae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rs74z" (UID: "27c60b28-4026-4737-a4f2-b69c08522dae") : secret "insights-runtime-extractor-tls" not found
Sep 15 11:25:41.471674 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.471509    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b887a34-bd19-4fa3-9897-a03605d5231c-metrics-tls\") pod \"dns-default-9wtp8\" (UID: \"8b887a34-bd19-4fa3-9897-a03605d5231c\") " pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:25:41.471674 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.471546    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/27c60b28-4026-4737-a4f2-b69c08522dae-data-volume\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:41.471674 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:41.471641    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 11:25:41.471674 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:41.471679    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b887a34-bd19-4fa3-9897-a03605d5231c-metrics-tls podName:8b887a34-bd19-4fa3-9897-a03605d5231c nodeName:}" failed. No retries permitted until 2026-09-15 11:25:45.471667795 +0000 UTC m=+44.134162959 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b887a34-bd19-4fa3-9897-a03605d5231c-metrics-tls") pod "dns-default-9wtp8" (UID: "8b887a34-bd19-4fa3-9897-a03605d5231c") : secret "dns-default-metrics-tls" not found
Sep 15 11:25:41.472127 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.472044    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/27c60b28-4026-4737-a4f2-b69c08522dae-data-volume\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:41.481448 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.481419    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqg5w\" (UniqueName: \"kubernetes.io/projected/27c60b28-4026-4737-a4f2-b69c08522dae-kube-api-access-lqg5w\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:41.975819 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.975773    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27c60b28-4026-4737-a4f2-b69c08522dae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:41.975999 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:41.975915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27c60b28-4026-4737-a4f2-b69c08522dae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:41.975999 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:41.975927    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 11:25:41.975999 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:41.975980    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 11:25:41.976205 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:41.976010    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27c60b28-4026-4737-a4f2-b69c08522dae-insights-runtime-extractor-tls podName:27c60b28-4026-4737-a4f2-b69c08522dae nodeName:}" failed. No retries permitted until 2026-09-15 11:25:42.975988911 +0000 UTC m=+41.638484093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27c60b28-4026-4737-a4f2-b69c08522dae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rs74z" (UID: "27c60b28-4026-4737-a4f2-b69c08522dae") : secret "insights-runtime-extractor-tls" not found
Sep 15 11:25:41.976205 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:41.976032    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27c60b28-4026-4737-a4f2-b69c08522dae-kube-rbac-proxy-cm podName:27c60b28-4026-4737-a4f2-b69c08522dae nodeName:}" failed. No retries permitted until 2026-09-15 11:25:42.976021512 +0000 UTC m=+41.638516670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/27c60b28-4026-4737-a4f2-b69c08522dae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rs74z" (UID: "27c60b28-4026-4737-a4f2-b69c08522dae") : configmap "kube-rbac-proxy" not found
Sep 15 11:25:42.313172 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.313075    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-tlpgr"]
Sep 15 11:25:42.349543 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.349499    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-tlpgr"]
Sep 15 11:25:42.349699 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.349647    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-tlpgr"
Sep 15 11:25:42.352036 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.352009    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 15 11:25:42.352432 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.352302    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 15 11:25:42.352547 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.352477    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wzc4h\""
Sep 15 11:25:42.479452 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.479411    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xm9h\" (UniqueName: \"kubernetes.io/projected/daf3093b-fc68-418c-ac0c-ccc1af1f007b-kube-api-access-6xm9h\") pod \"migrator-8b4d697c5-tlpgr\" (UID: \"daf3093b-fc68-418c-ac0c-ccc1af1f007b\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-tlpgr"
Sep 15 11:25:42.580297 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.580223    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xm9h\" (UniqueName: \"kubernetes.io/projected/daf3093b-fc68-418c-ac0c-ccc1af1f007b-kube-api-access-6xm9h\") pod \"migrator-8b4d697c5-tlpgr\" (UID: \"daf3093b-fc68-418c-ac0c-ccc1af1f007b\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-tlpgr"
Sep 15 11:25:42.588069 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.588035    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xm9h\" (UniqueName: \"kubernetes.io/projected/daf3093b-fc68-418c-ac0c-ccc1af1f007b-kube-api-access-6xm9h\") pod \"migrator-8b4d697c5-tlpgr\" (UID: \"daf3093b-fc68-418c-ac0c-ccc1af1f007b\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-tlpgr"
Sep 15 11:25:42.661072 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.661021    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-tlpgr"
Sep 15 11:25:42.679462 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.679431    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86d58d4fdf-q72nw"]
Sep 15 11:25:42.693935 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.693896    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.694480 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.694461    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86d58d4fdf-q72nw"]
Sep 15 11:25:42.696305 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.696269    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xsd55\""
Sep 15 11:25:42.696411 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.696305    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 11:25:42.696411 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.696350    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 11:25:42.696591 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.696575    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 11:25:42.700623 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.700602    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 11:25:42.782817 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.782782    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7547112-c429-4178-a48e-f47e6ef7576c-installation-pull-secrets\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.782975 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.782843    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.782975 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.782919    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7547112-c429-4178-a48e-f47e6ef7576c-trusted-ca\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.782975 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.782946    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thfdf\" (UniqueName: \"kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-kube-api-access-thfdf\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.782975 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.782972    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7547112-c429-4178-a48e-f47e6ef7576c-ca-trust-extracted\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.783157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.782990    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7547112-c429-4178-a48e-f47e6ef7576c-registry-certificates\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.783157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.783034    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7547112-c429-4178-a48e-f47e6ef7576c-image-registry-private-configuration\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.783157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.783114    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-bound-sa-token\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.817225 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.817190    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-whk78"]
Sep 15 11:25:42.830461 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.830392    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f5l6r_dbb4a488-5113-44e5-bea0-b831896d7cac/dns-node-resolver/0.log"
Sep 15 11:25:42.837550 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.837531    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-whk78"]
Sep 15 11:25:42.837655 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.837617    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-whk78"
Sep 15 11:25:42.839678 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.839649    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-cn7sv\""
Sep 15 11:25:42.883539 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.883508    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.883673 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.883589    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7547112-c429-4178-a48e-f47e6ef7576c-trusted-ca\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.883673 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.883620    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thfdf\" (UniqueName: \"kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-kube-api-access-thfdf\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.883789 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:42.883739    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 11:25:42.883789 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:42.883764    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86d58d4fdf-q72nw: secret "image-registry-tls" not found
Sep 15 11:25:42.883897 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.883783    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7547112-c429-4178-a48e-f47e6ef7576c-ca-trust-extracted\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.883897 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:42.883826    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls podName:f7547112-c429-4178-a48e-f47e6ef7576c nodeName:}" failed. No retries permitted until 2026-09-15 11:25:43.383805535 +0000 UTC m=+42.046300724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls") pod "image-registry-86d58d4fdf-q72nw" (UID: "f7547112-c429-4178-a48e-f47e6ef7576c") : secret "image-registry-tls" not found
Sep 15 11:25:42.883897 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.883864    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7547112-c429-4178-a48e-f47e6ef7576c-registry-certificates\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.884114 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.883913    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7547112-c429-4178-a48e-f47e6ef7576c-image-registry-private-configuration\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.884114 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.883965    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-bound-sa-token\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.884114 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.884005    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7547112-c429-4178-a48e-f47e6ef7576c-installation-pull-secrets\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.884487 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.884462    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7547112-c429-4178-a48e-f47e6ef7576c-registry-certificates\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.884698 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.884676    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7547112-c429-4178-a48e-f47e6ef7576c-trusted-ca\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.885132 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.885111    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7547112-c429-4178-a48e-f47e6ef7576c-ca-trust-extracted\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.886877 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.886853    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7547112-c429-4178-a48e-f47e6ef7576c-image-registry-private-configuration\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.886877 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.886864    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7547112-c429-4178-a48e-f47e6ef7576c-installation-pull-secrets\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.894470 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.894423    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-bound-sa-token\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.895243 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.895222    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thfdf\" (UniqueName: \"kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-kube-api-access-thfdf\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:42.985017 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.984993    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27c60b28-4026-4737-a4f2-b69c08522dae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:42.985133 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.985027    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27c60b28-4026-4737-a4f2-b69c08522dae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:42.985133 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:42.985074    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l55d\" (UniqueName: \"kubernetes.io/projected/9f51c635-65cd-4976-8d82-cb6dbf7797a2-kube-api-access-8l55d\") pod \"network-check-source-6f8b768455-whk78\" (UID: \"9f51c635-65cd-4976-8d82-cb6dbf7797a2\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-whk78"
Sep 15 11:25:42.985295 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:42.985133    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 11:25:42.985295 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:42.985166    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 11:25:42.985295 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:42.985208    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27c60b28-4026-4737-a4f2-b69c08522dae-kube-rbac-proxy-cm podName:27c60b28-4026-4737-a4f2-b69c08522dae nodeName:}" failed. No retries permitted until 2026-09-15 11:25:44.985186578 +0000 UTC m=+43.647681756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/27c60b28-4026-4737-a4f2-b69c08522dae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rs74z" (UID: "27c60b28-4026-4737-a4f2-b69c08522dae") : configmap "kube-rbac-proxy" not found
Sep 15 11:25:42.985295 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:42.985227    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27c60b28-4026-4737-a4f2-b69c08522dae-insights-runtime-extractor-tls podName:27c60b28-4026-4737-a4f2-b69c08522dae nodeName:}" failed. No retries permitted until 2026-09-15 11:25:44.985218294 +0000 UTC m=+43.647713453 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27c60b28-4026-4737-a4f2-b69c08522dae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rs74z" (UID: "27c60b28-4026-4737-a4f2-b69c08522dae") : secret "insights-runtime-extractor-tls" not found
Sep 15 11:25:43.086149 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.085993    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l55d\" (UniqueName: \"kubernetes.io/projected/9f51c635-65cd-4976-8d82-cb6dbf7797a2-kube-api-access-8l55d\") pod \"network-check-source-6f8b768455-whk78\" (UID: \"9f51c635-65cd-4976-8d82-cb6dbf7797a2\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-whk78"
Sep 15 11:25:43.096644 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.096614    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l55d\" (UniqueName: \"kubernetes.io/projected/9f51c635-65cd-4976-8d82-cb6dbf7797a2-kube-api-access-8l55d\") pod \"network-check-source-6f8b768455-whk78\" (UID: \"9f51c635-65cd-4976-8d82-cb6dbf7797a2\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-whk78"
Sep 15 11:25:43.100872 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.100843    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-tlpgr"]
Sep 15 11:25:43.104162 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:43.104128    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf3093b_fc68_418c_ac0c_ccc1af1f007b.slice/crio-40429c93d91d143b360c2b70adaf667be6ddf11427b8ebf1ffaa8ad3f39fc116 WatchSource:0}: Error finding container 40429c93d91d143b360c2b70adaf667be6ddf11427b8ebf1ffaa8ad3f39fc116: Status 404 returned error can't find the container with id 40429c93d91d143b360c2b70adaf667be6ddf11427b8ebf1ffaa8ad3f39fc116
Sep 15 11:25:43.153576 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.153542    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-whk78"
Sep 15 11:25:43.171121 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.171087    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tgsbr" event={"ID":"356a69de-a242-4342-87a5-f876c681dd1e","Type":"ContainerStarted","Data":"60c1fdd147989b803c1be1d0c52b3b610b2cc5a20e1ae02cfc1f7a426c74ab09"}
Sep 15 11:25:43.172394 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.172366    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-tlpgr" event={"ID":"daf3093b-fc68-418c-ac0c-ccc1af1f007b","Type":"ContainerStarted","Data":"40429c93d91d143b360c2b70adaf667be6ddf11427b8ebf1ffaa8ad3f39fc116"}
Sep 15 11:25:43.189151 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.189102    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tgsbr" podStartSLOduration=1.875148259 podStartE2EDuration="6.189078589s" totalImagesPullingTime="4.31393033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:43.188294255 +0000 UTC m=+41.850789456" watchObservedRunningTime="2026-09-15 11:25:43.189078589 +0000 UTC m=+41.851573769"
Sep 15 11:25:43.388795 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.388702    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:43.389171 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:43.388861    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 11:25:43.389171 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:43.388879    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86d58d4fdf-q72nw: secret "image-registry-tls" not found
Sep 15 11:25:43.389171 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:43.388936    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls podName:f7547112-c429-4178-a48e-f47e6ef7576c nodeName:}" failed. No retries permitted until 2026-09-15 11:25:44.388917887 +0000 UTC m=+43.051413053 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls") pod "image-registry-86d58d4fdf-q72nw" (UID: "f7547112-c429-4178-a48e-f47e6ef7576c") : secret "image-registry-tls" not found
Sep 15 11:25:43.431630 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.431605    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-42tpl_85bb9ddc-8249-4923-a96d-4b17b5450c51/node-ca/0.log"
Sep 15 11:25:43.454585 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.454553    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-whk78"]
Sep 15 11:25:43.458406 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:43.458376    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f51c635_65cd_4976_8d82_cb6dbf7797a2.slice/crio-17d81328521cfca669aea29f41558c2b0c06686fd4726258cca71f037eb5d0b3 WatchSource:0}: Error finding container 17d81328521cfca669aea29f41558c2b0c06686fd4726258cca71f037eb5d0b3: Status 404 returned error can't find the container with id 17d81328521cfca669aea29f41558c2b0c06686fd4726258cca71f037eb5d0b3
Sep 15 11:25:43.761583 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.761506    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-mh6qf"]
Sep 15 11:25:43.792439 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.792411    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-mh6qf"]
Sep 15 11:25:43.792579 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.792507    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-mh6qf"
Sep 15 11:25:43.794731 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.794708    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 15 11:25:43.794845 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.794740    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 15 11:25:43.794845 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.794814    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9thbt\""
Sep 15 11:25:43.794845 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.794827    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 15 11:25:43.794998 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.794873    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 15 11:25:43.893539 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.893477    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8737bd2b-0179-42d6-812f-82d30b14c7ad-signing-key\") pod \"service-ca-568869fdc5-mh6qf\" (UID: \"8737bd2b-0179-42d6-812f-82d30b14c7ad\") " pod="openshift-service-ca/service-ca-568869fdc5-mh6qf"
Sep 15 11:25:43.893731 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.893548    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8737bd2b-0179-42d6-812f-82d30b14c7ad-signing-cabundle\") pod \"service-ca-568869fdc5-mh6qf\" (UID: \"8737bd2b-0179-42d6-812f-82d30b14c7ad\") " pod="openshift-service-ca/service-ca-568869fdc5-mh6qf"
Sep 15 11:25:43.893731 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.893714    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mltzt\" (UniqueName: \"kubernetes.io/projected/8737bd2b-0179-42d6-812f-82d30b14c7ad-kube-api-access-mltzt\") pod \"service-ca-568869fdc5-mh6qf\" (UID: \"8737bd2b-0179-42d6-812f-82d30b14c7ad\") " pod="openshift-service-ca/service-ca-568869fdc5-mh6qf"
Sep 15 11:25:43.994695 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.994653    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8737bd2b-0179-42d6-812f-82d30b14c7ad-signing-key\") pod \"service-ca-568869fdc5-mh6qf\" (UID: \"8737bd2b-0179-42d6-812f-82d30b14c7ad\") " pod="openshift-service-ca/service-ca-568869fdc5-mh6qf"
Sep 15 11:25:43.994695 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.994700    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8737bd2b-0179-42d6-812f-82d30b14c7ad-signing-cabundle\") pod \"service-ca-568869fdc5-mh6qf\" (UID: \"8737bd2b-0179-42d6-812f-82d30b14c7ad\") " pod="openshift-service-ca/service-ca-568869fdc5-mh6qf"
Sep 15 11:25:43.994945 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.994817    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mltzt\" (UniqueName: \"kubernetes.io/projected/8737bd2b-0179-42d6-812f-82d30b14c7ad-kube-api-access-mltzt\") pod \"service-ca-568869fdc5-mh6qf\" (UID: \"8737bd2b-0179-42d6-812f-82d30b14c7ad\") " pod="openshift-service-ca/service-ca-568869fdc5-mh6qf"
Sep 15 11:25:43.995503 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.995480    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8737bd2b-0179-42d6-812f-82d30b14c7ad-signing-cabundle\") pod \"service-ca-568869fdc5-mh6qf\" (UID: \"8737bd2b-0179-42d6-812f-82d30b14c7ad\") " pod="openshift-service-ca/service-ca-568869fdc5-mh6qf"
Sep 15 11:25:43.997724 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:43.997693    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8737bd2b-0179-42d6-812f-82d30b14c7ad-signing-key\") pod \"service-ca-568869fdc5-mh6qf\" (UID: \"8737bd2b-0179-42d6-812f-82d30b14c7ad\") " pod="openshift-service-ca/service-ca-568869fdc5-mh6qf"
Sep 15 11:25:44.003312 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:44.003288    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mltzt\" (UniqueName: \"kubernetes.io/projected/8737bd2b-0179-42d6-812f-82d30b14c7ad-kube-api-access-mltzt\") pod \"service-ca-568869fdc5-mh6qf\" (UID: \"8737bd2b-0179-42d6-812f-82d30b14c7ad\") " pod="openshift-service-ca/service-ca-568869fdc5-mh6qf"
Sep 15 11:25:44.101482 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:44.101451    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-mh6qf"
Sep 15 11:25:44.176554 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:44.176517    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-whk78" event={"ID":"9f51c635-65cd-4976-8d82-cb6dbf7797a2","Type":"ContainerStarted","Data":"17d81328521cfca669aea29f41558c2b0c06686fd4726258cca71f037eb5d0b3"}
Sep 15 11:25:44.398611 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:44.398524    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:44.399001 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:44.398670    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 11:25:44.399001 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:44.398690    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86d58d4fdf-q72nw: secret "image-registry-tls" not found
Sep 15 11:25:44.399001 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:44.398753    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls podName:f7547112-c429-4178-a48e-f47e6ef7576c nodeName:}" failed. No retries permitted until 2026-09-15 11:25:46.398732982 +0000 UTC m=+45.061228156 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls") pod "image-registry-86d58d4fdf-q72nw" (UID: "f7547112-c429-4178-a48e-f47e6ef7576c") : secret "image-registry-tls" not found
Sep 15 11:25:44.404171 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:44.404099    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-mh6qf"]
Sep 15 11:25:44.407564 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:44.407540    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8737bd2b_0179_42d6_812f_82d30b14c7ad.slice/crio-30dda808763186e299ac280a4d7b0b63b4ecf69774e7c147ab0a13817a4fb257 WatchSource:0}: Error finding container 30dda808763186e299ac280a4d7b0b63b4ecf69774e7c147ab0a13817a4fb257: Status 404 returned error can't find the container with id 30dda808763186e299ac280a4d7b0b63b4ecf69774e7c147ab0a13817a4fb257
Sep 15 11:25:44.431438 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:44.431416    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tgsbr_356a69de-a242-4342-87a5-f876c681dd1e/serve-healthcheck-canary/0.log"
Sep 15 11:25:45.003730 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:45.003697    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27c60b28-4026-4737-a4f2-b69c08522dae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:45.004071 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:45.003746    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27c60b28-4026-4737-a4f2-b69c08522dae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:45.004071 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:45.003841    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 11:25:45.004071 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:45.003913    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27c60b28-4026-4737-a4f2-b69c08522dae-kube-rbac-proxy-cm podName:27c60b28-4026-4737-a4f2-b69c08522dae nodeName:}" failed. No retries permitted until 2026-09-15 11:25:49.003894533 +0000 UTC m=+47.666389696 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/27c60b28-4026-4737-a4f2-b69c08522dae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rs74z" (UID: "27c60b28-4026-4737-a4f2-b69c08522dae") : configmap "kube-rbac-proxy" not found
Sep 15 11:25:45.004071 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:45.003952    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 11:25:45.004071 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:45.004015    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27c60b28-4026-4737-a4f2-b69c08522dae-insights-runtime-extractor-tls podName:27c60b28-4026-4737-a4f2-b69c08522dae nodeName:}" failed. No retries permitted until 2026-09-15 11:25:49.003995905 +0000 UTC m=+47.666491079 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27c60b28-4026-4737-a4f2-b69c08522dae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rs74z" (UID: "27c60b28-4026-4737-a4f2-b69c08522dae") : secret "insights-runtime-extractor-tls" not found
Sep 15 11:25:45.180535 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:45.180502    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-mh6qf" event={"ID":"8737bd2b-0179-42d6-812f-82d30b14c7ad","Type":"ContainerStarted","Data":"30dda808763186e299ac280a4d7b0b63b4ecf69774e7c147ab0a13817a4fb257"}
Sep 15 11:25:45.507537 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:45.507500    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b887a34-bd19-4fa3-9897-a03605d5231c-metrics-tls\") pod \"dns-default-9wtp8\" (UID: \"8b887a34-bd19-4fa3-9897-a03605d5231c\") " pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:25:45.507990 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:45.507650    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 11:25:45.507990 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:45.507717    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b887a34-bd19-4fa3-9897-a03605d5231c-metrics-tls podName:8b887a34-bd19-4fa3-9897-a03605d5231c nodeName:}" failed. No retries permitted until 2026-09-15 11:25:53.507696387 +0000 UTC m=+52.170191545 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b887a34-bd19-4fa3-9897-a03605d5231c-metrics-tls") pod "dns-default-9wtp8" (UID: "8b887a34-bd19-4fa3-9897-a03605d5231c") : secret "dns-default-metrics-tls" not found
Sep 15 11:25:46.415294 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:46.415254    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:46.415469 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:46.415430    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 11:25:46.415469 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:46.415458    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86d58d4fdf-q72nw: secret "image-registry-tls" not found
Sep 15 11:25:46.415542 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:46.415517    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls podName:f7547112-c429-4178-a48e-f47e6ef7576c nodeName:}" failed. No retries permitted until 2026-09-15 11:25:50.415502499 +0000 UTC m=+49.077997658 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls") pod "image-registry-86d58d4fdf-q72nw" (UID: "f7547112-c429-4178-a48e-f47e6ef7576c") : secret "image-registry-tls" not found
Sep 15 11:25:47.186913 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:47.186875    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-whk78" event={"ID":"9f51c635-65cd-4976-8d82-cb6dbf7797a2","Type":"ContainerStarted","Data":"ca6103b5b1274a630c4bd4c68a4d16ebe82058ba684bda7c66fcbe3a28b99624"}
Sep 15 11:25:47.202162 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:47.202101    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-whk78" podStartSLOduration=1.9689947650000001 podStartE2EDuration="5.202029064s" totalImagesPullingTime="3.233034299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:47.200388924 +0000 UTC m=+45.862884124" watchObservedRunningTime="2026-09-15 11:25:47.202029064 +0000 UTC m=+45.864524248"
Sep 15 11:25:49.035784 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:49.035677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27c60b28-4026-4737-a4f2-b69c08522dae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:49.035784 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:49.035728    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27c60b28-4026-4737-a4f2-b69c08522dae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:49.036346 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:49.035880    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 11:25:49.036346 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:49.035933    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27c60b28-4026-4737-a4f2-b69c08522dae-insights-runtime-extractor-tls podName:27c60b28-4026-4737-a4f2-b69c08522dae nodeName:}" failed. No retries permitted until 2026-09-15 11:25:57.035919468 +0000 UTC m=+55.698414640 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27c60b28-4026-4737-a4f2-b69c08522dae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rs74z" (UID: "27c60b28-4026-4737-a4f2-b69c08522dae") : secret "insights-runtime-extractor-tls" not found
Sep 15 11:25:49.036346 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:49.036248    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27c60b28-4026-4737-a4f2-b69c08522dae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:49.194102 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:49.194051    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-tlpgr" event={"ID":"daf3093b-fc68-418c-ac0c-ccc1af1f007b","Type":"ContainerStarted","Data":"422a9e9e5eb7769949b5e8860c3ad54ddf0b1ae1c6dcf669dd3472662bba35f8"}
Sep 15 11:25:49.194301 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:49.194131    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-tlpgr" event={"ID":"daf3093b-fc68-418c-ac0c-ccc1af1f007b","Type":"ContainerStarted","Data":"62d38ff49880e51627187e6a1b4e3a1c814d19dae4d5c12a4594141f62dd8ac2"}
Sep 15 11:25:49.195613 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:49.195585    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-mh6qf" event={"ID":"8737bd2b-0179-42d6-812f-82d30b14c7ad","Type":"ContainerStarted","Data":"ff587889972e22aac602f3ccb89ac387b835b4350c5111227e5670ccba9ad058"}
Sep 15 11:25:49.209467 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:49.209418    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-tlpgr" podStartSLOduration=1.961660829 podStartE2EDuration="7.209400192s" totalImagesPullingTime="5.247739363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:49.208032609 +0000 UTC m=+47.870527802" watchObservedRunningTime="2026-09-15 11:25:49.209400192 +0000 UTC m=+47.871895376"
Sep 15 11:25:49.234646 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:49.234599    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-mh6qf" podStartSLOduration=1.9022838549999999 podStartE2EDuration="6.234583896s" totalImagesPullingTime="4.332300041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:49.233630307 +0000 UTC m=+47.896125507" watchObservedRunningTime="2026-09-15 11:25:49.234583896 +0000 UTC m=+47.897079077"
Sep 15 11:25:50.448457 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:50.448419    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:50.448941 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:50.448621    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 11:25:50.448941 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:50.448649    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86d58d4fdf-q72nw: secret "image-registry-tls" not found
Sep 15 11:25:50.448941 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:25:50.448723    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls podName:f7547112-c429-4178-a48e-f47e6ef7576c nodeName:}" failed. No retries permitted until 2026-09-15 11:25:58.448701622 +0000 UTC m=+57.111196790 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls") pod "image-registry-86d58d4fdf-q72nw" (UID: "f7547112-c429-4178-a48e-f47e6ef7576c") : secret "image-registry-tls" not found
Sep 15 11:25:53.572855 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:53.572822    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b887a34-bd19-4fa3-9897-a03605d5231c-metrics-tls\") pod \"dns-default-9wtp8\" (UID: \"8b887a34-bd19-4fa3-9897-a03605d5231c\") " pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:25:53.575231 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:53.575199    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b887a34-bd19-4fa3-9897-a03605d5231c-metrics-tls\") pod \"dns-default-9wtp8\" (UID: \"8b887a34-bd19-4fa3-9897-a03605d5231c\") " pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:25:53.643193 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:53.643156    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:25:53.930026 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:53.929992    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9wtp8"]
Sep 15 11:25:53.933770 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:53.933744    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b887a34_bd19_4fa3_9897_a03605d5231c.slice/crio-867c0e2e7c54bbd567c0636bca997db302482ef636cf1cec0b23ec471194ebf2 WatchSource:0}: Error finding container 867c0e2e7c54bbd567c0636bca997db302482ef636cf1cec0b23ec471194ebf2: Status 404 returned error can't find the container with id 867c0e2e7c54bbd567c0636bca997db302482ef636cf1cec0b23ec471194ebf2
Sep 15 11:25:54.210383 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:54.210296    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9wtp8" event={"ID":"8b887a34-bd19-4fa3-9897-a03605d5231c","Type":"ContainerStarted","Data":"867c0e2e7c54bbd567c0636bca997db302482ef636cf1cec0b23ec471194ebf2"}
Sep 15 11:25:56.219422 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:56.219379    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9wtp8" event={"ID":"8b887a34-bd19-4fa3-9897-a03605d5231c","Type":"ContainerStarted","Data":"0e441f36c0d5d56dd2f6eb793008fbefcd96aea88a1ba06f89e927934a0367d0"}
Sep 15 11:25:56.219843 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:56.219439    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9wtp8" event={"ID":"8b887a34-bd19-4fa3-9897-a03605d5231c","Type":"ContainerStarted","Data":"6a3eb06e5a56959aeaf6ed67e5898415343dd83650d74cf3e34147a879cdedfa"}
Sep 15 11:25:56.219843 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:56.219543    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:25:56.237199 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:56.237143    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9wtp8" podStartSLOduration=17.651622445 podStartE2EDuration="19.237129435s" totalImagesPullingTime="1.58550699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:56.236832904 +0000 UTC m=+54.899328084" watchObservedRunningTime="2026-09-15 11:25:56.237129435 +0000 UTC m=+54.899624651"
Sep 15 11:25:57.099211 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:57.099153    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27c60b28-4026-4737-a4f2-b69c08522dae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:57.101440 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:57.101413    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27c60b28-4026-4737-a4f2-b69c08522dae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rs74z\" (UID: \"27c60b28-4026-4737-a4f2-b69c08522dae\") " pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:57.192758 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:57.192725    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rs74z"
Sep 15 11:25:57.464546 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:57.464428    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rs74z"]
Sep 15 11:25:57.468179 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:57.468147    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c60b28_4026_4737_a4f2_b69c08522dae.slice/crio-40ed3e88125f2ca25c93f90377aaf35a7934ae662eb67cd33fa4923af1219f86 WatchSource:0}: Error finding container 40ed3e88125f2ca25c93f90377aaf35a7934ae662eb67cd33fa4923af1219f86: Status 404 returned error can't find the container with id 40ed3e88125f2ca25c93f90377aaf35a7934ae662eb67cd33fa4923af1219f86
Sep 15 11:25:58.230160 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:58.230123    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rs74z" event={"ID":"27c60b28-4026-4737-a4f2-b69c08522dae","Type":"ContainerStarted","Data":"803790bad1ab978d76a3a3bbe447c0e9dfabe20bdb163aef6db839b12c8db41d"}
Sep 15 11:25:58.230325 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:58.230182    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rs74z" event={"ID":"27c60b28-4026-4737-a4f2-b69c08522dae","Type":"ContainerStarted","Data":"405ada6b5cb8d70a2a1bec1c38e33cfdbd06b6925fedaaba4a1c18ef1f607759"}
Sep 15 11:25:58.230325 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:58.230201    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rs74z" event={"ID":"27c60b28-4026-4737-a4f2-b69c08522dae","Type":"ContainerStarted","Data":"40ed3e88125f2ca25c93f90377aaf35a7934ae662eb67cd33fa4923af1219f86"}
Sep 15 11:25:58.511810 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:58.511766    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:58.514852 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:58.514825    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls\") pod \"image-registry-86d58d4fdf-q72nw\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") " pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:58.603924 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:58.603897    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:58.890665 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:58.890633    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86d58d4fdf-q72nw"]
Sep 15 11:25:58.892667 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:25:58.892633    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7547112_c429_4178_a48e_f47e6ef7576c.slice/crio-3771ce37aa4eed6ce7403a5a03d431d49fcbb139b6198a33aa6adcecbdd0c95d WatchSource:0}: Error finding container 3771ce37aa4eed6ce7403a5a03d431d49fcbb139b6198a33aa6adcecbdd0c95d: Status 404 returned error can't find the container with id 3771ce37aa4eed6ce7403a5a03d431d49fcbb139b6198a33aa6adcecbdd0c95d
Sep 15 11:25:59.236968 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:59.236852    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw" event={"ID":"f7547112-c429-4178-a48e-f47e6ef7576c","Type":"ContainerStarted","Data":"608404d21c4c6a71280256ec4bbc78f0c0e1ea634b49acf611343896d661913c"}
Sep 15 11:25:59.236968 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:59.236945    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw" event={"ID":"f7547112-c429-4178-a48e-f47e6ef7576c","Type":"ContainerStarted","Data":"3771ce37aa4eed6ce7403a5a03d431d49fcbb139b6198a33aa6adcecbdd0c95d"}
Sep 15 11:25:59.237239 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:59.237215    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:25:59.261120 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:25:59.261042    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw" podStartSLOduration=17.261022871 podStartE2EDuration="17.261022871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:59.260205805 +0000 UTC m=+57.922701016" watchObservedRunningTime="2026-09-15 11:25:59.261022871 +0000 UTC m=+57.923518055"
Sep 15 11:26:00.242502 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:00.242468    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rs74z" event={"ID":"27c60b28-4026-4737-a4f2-b69c08522dae","Type":"ContainerStarted","Data":"d3ea08ba6522c0abf9326b00bbbdb97ea21736fe59a6e7318f66f7cd3c80d20c"}
Sep 15 11:26:00.264926 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:00.264866    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rs74z" podStartSLOduration=17.445474034 podStartE2EDuration="19.264845867s" totalImagesPullingTime="1.819371833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:00.264366108 +0000 UTC m=+58.926861299" watchObservedRunningTime="2026-09-15 11:26:00.264845867 +0000 UTC m=+58.927341049"
Sep 15 11:26:03.140461 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:03.140428    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b8qtl"
Sep 15 11:26:06.227783 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.227754    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9wtp8"
Sep 15 11:26:06.663642 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.663601    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-p6cx4"]
Sep 15 11:26:06.667499 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.667474    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6cx4"
Sep 15 11:26:06.669578 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.669553    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-k4g76\""
Sep 15 11:26:06.670030 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.670013    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 15 11:26:06.670146 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.670131    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 15 11:26:06.675013 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.674990    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f6487989-8xxx8"]
Sep 15 11:26:06.676588 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.676572    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bfccd9fd-ppnrc"]
Sep 15 11:26:06.676697 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.676683    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f6487989-8xxx8"
Sep 15 11:26:06.678654 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.678627    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bfccd9fd-ppnrc"
Sep 15 11:26:06.678783 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.678766    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"]
Sep 15 11:26:06.678879 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.678788    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 15 11:26:06.679513 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.679495    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 15 11:26:06.679778 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.679755    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 15 11:26:06.680499 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.680480    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 15 11:26:06.681046 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.681032    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-p6cx4"]
Sep 15 11:26:06.681142 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.681131    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"
Sep 15 11:26:06.681581 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.681563    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-77j6s\""
Sep 15 11:26:06.683340 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.683322    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 15 11:26:06.683450 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.683377    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 15 11:26:06.683450 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.683398    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 15 11:26:06.684157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.684138    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 15 11:26:06.684748 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.684730    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 15 11:26:06.692393 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.692368    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bfccd9fd-ppnrc"]
Sep 15 11:26:06.693235 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.693204    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f6487989-8xxx8"]
Sep 15 11:26:06.696430 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.696408    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"]
Sep 15 11:26:06.701173 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.701154    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86d58d4fdf-q72nw"]
Sep 15 11:26:06.763096 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.763033    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-759494458-fl68h"]
Sep 15 11:26:06.764931 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.764916    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.771197 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.771166    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg2pr\" (UniqueName: \"kubernetes.io/projected/a3fcdb73-b50f-41bb-a829-551da06facff-kube-api-access-kg2pr\") pod \"klusterlet-addon-workmgr-59f6487989-8xxx8\" (UID: \"a3fcdb73-b50f-41bb-a829-551da06facff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f6487989-8xxx8"
Sep 15 11:26:06.771334 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.771207    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7afafd1c-e60e-447f-8d8a-5441a20af1ec-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-57bfccd9fd-ppnrc\" (UID: \"7afafd1c-e60e-447f-8d8a-5441a20af1ec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bfccd9fd-ppnrc"
Sep 15 11:26:06.771334 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.771239    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkkwh\" (UniqueName: \"kubernetes.io/projected/90992df9-ddc5-413d-93c6-5cb6c9cd5038-kube-api-access-tkkwh\") pod \"cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5\" (UID: \"90992df9-ddc5-413d-93c6-5cb6c9cd5038\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"
Sep 15 11:26:06.771334 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.771303    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90992df9-ddc5-413d-93c6-5cb6c9cd5038-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5\" (UID: \"90992df9-ddc5-413d-93c6-5cb6c9cd5038\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"
Sep 15 11:26:06.771501 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.771378    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3fcdb73-b50f-41bb-a829-551da06facff-tmp\") pod \"klusterlet-addon-workmgr-59f6487989-8xxx8\" (UID: \"a3fcdb73-b50f-41bb-a829-551da06facff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f6487989-8xxx8"
Sep 15 11:26:06.771501 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.771431    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnh9w\" (UniqueName: \"kubernetes.io/projected/7afafd1c-e60e-447f-8d8a-5441a20af1ec-kube-api-access-qnh9w\") pod \"managed-serviceaccount-addon-agent-57bfccd9fd-ppnrc\" (UID: \"7afafd1c-e60e-447f-8d8a-5441a20af1ec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bfccd9fd-ppnrc"
Sep 15 11:26:06.771608 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.771519    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b085c684-f385-4e33-b780-492f1aac71b8-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-p6cx4\" (UID: \"b085c684-f385-4e33-b780-492f1aac71b8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6cx4"
Sep 15 11:26:06.771608 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.771553    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b085c684-f385-4e33-b780-492f1aac71b8-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-p6cx4\" (UID: \"b085c684-f385-4e33-b780-492f1aac71b8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6cx4"
Sep 15 11:26:06.771608 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.771587    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/90992df9-ddc5-413d-93c6-5cb6c9cd5038-hub\") pod \"cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5\" (UID: \"90992df9-ddc5-413d-93c6-5cb6c9cd5038\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"
Sep 15 11:26:06.771756 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.771613    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/90992df9-ddc5-413d-93c6-5cb6c9cd5038-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5\" (UID: \"90992df9-ddc5-413d-93c6-5cb6c9cd5038\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"
Sep 15 11:26:06.771756 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.771653    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a3fcdb73-b50f-41bb-a829-551da06facff-klusterlet-config\") pod \"klusterlet-addon-workmgr-59f6487989-8xxx8\" (UID: \"a3fcdb73-b50f-41bb-a829-551da06facff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f6487989-8xxx8"
Sep 15 11:26:06.771756 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.771677    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/90992df9-ddc5-413d-93c6-5cb6c9cd5038-ca\") pod \"cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5\" (UID: \"90992df9-ddc5-413d-93c6-5cb6c9cd5038\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"
Sep 15 11:26:06.771756 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.771712    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/90992df9-ddc5-413d-93c6-5cb6c9cd5038-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5\" (UID: \"90992df9-ddc5-413d-93c6-5cb6c9cd5038\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"
Sep 15 11:26:06.788095 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.788047    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-759494458-fl68h"]
Sep 15 11:26:06.872410 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.872360    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-ca-trust-extracted\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.872410 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.872409    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-registry-tls\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.872651 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.872439    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-image-registry-private-configuration\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.872651 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.872475    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b085c684-f385-4e33-b780-492f1aac71b8-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-p6cx4\" (UID: \"b085c684-f385-4e33-b780-492f1aac71b8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6cx4"
Sep 15 11:26:06.872651 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.872506    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b085c684-f385-4e33-b780-492f1aac71b8-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-p6cx4\" (UID: \"b085c684-f385-4e33-b780-492f1aac71b8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6cx4"
Sep 15 11:26:06.872651 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.872534    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-installation-pull-secrets\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.872651 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.872557    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-bound-sa-token\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.872651 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.872586    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/90992df9-ddc5-413d-93c6-5cb6c9cd5038-hub\") pod \"cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5\" (UID: \"90992df9-ddc5-413d-93c6-5cb6c9cd5038\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"
Sep 15 11:26:06.872651 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.872613    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/90992df9-ddc5-413d-93c6-5cb6c9cd5038-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5\" (UID: \"90992df9-ddc5-413d-93c6-5cb6c9cd5038\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"
Sep 15 11:26:06.872986 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.872654    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-trusted-ca\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.872986 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.872840    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a3fcdb73-b50f-41bb-a829-551da06facff-klusterlet-config\") pod \"klusterlet-addon-workmgr-59f6487989-8xxx8\" (UID: \"a3fcdb73-b50f-41bb-a829-551da06facff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f6487989-8xxx8"
Sep 15 11:26:06.872986 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.872870    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/90992df9-ddc5-413d-93c6-5cb6c9cd5038-ca\") pod \"cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5\" (UID: \"90992df9-ddc5-413d-93c6-5cb6c9cd5038\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"
Sep 15 11:26:06.872986 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.872900    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/90992df9-ddc5-413d-93c6-5cb6c9cd5038-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5\" (UID: \"90992df9-ddc5-413d-93c6-5cb6c9cd5038\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"
Sep 15 11:26:06.872986 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.872941    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-registry-certificates\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.872986 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.872976    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg2pr\" (UniqueName: \"kubernetes.io/projected/a3fcdb73-b50f-41bb-a829-551da06facff-kube-api-access-kg2pr\") pod \"klusterlet-addon-workmgr-59f6487989-8xxx8\" (UID: \"a3fcdb73-b50f-41bb-a829-551da06facff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f6487989-8xxx8"
Sep 15 11:26:06.873319 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.873307    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7afafd1c-e60e-447f-8d8a-5441a20af1ec-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-57bfccd9fd-ppnrc\" (UID: \"7afafd1c-e60e-447f-8d8a-5441a20af1ec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bfccd9fd-ppnrc"
Sep 15 11:26:06.873376 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.873345    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkkwh\" (UniqueName: \"kubernetes.io/projected/90992df9-ddc5-413d-93c6-5cb6c9cd5038-kube-api-access-tkkwh\") pod \"cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5\" (UID: \"90992df9-ddc5-413d-93c6-5cb6c9cd5038\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"
Sep 15 11:26:06.873433 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.873372    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90992df9-ddc5-413d-93c6-5cb6c9cd5038-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5\" (UID: \"90992df9-ddc5-413d-93c6-5cb6c9cd5038\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"
Sep 15 11:26:06.873433 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.873415    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3fcdb73-b50f-41bb-a829-551da06facff-tmp\") pod \"klusterlet-addon-workmgr-59f6487989-8xxx8\" (UID: \"a3fcdb73-b50f-41bb-a829-551da06facff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f6487989-8xxx8"
Sep 15 11:26:06.873537 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.873445    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2kdh\" (UniqueName: \"kubernetes.io/projected/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-kube-api-access-q2kdh\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.873537 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.873485    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnh9w\" (UniqueName: \"kubernetes.io/projected/7afafd1c-e60e-447f-8d8a-5441a20af1ec-kube-api-access-qnh9w\") pod \"managed-serviceaccount-addon-agent-57bfccd9fd-ppnrc\" (UID: \"7afafd1c-e60e-447f-8d8a-5441a20af1ec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bfccd9fd-ppnrc"
Sep 15 11:26:06.874246 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.873706    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/90992df9-ddc5-413d-93c6-5cb6c9cd5038-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5\" (UID: \"90992df9-ddc5-413d-93c6-5cb6c9cd5038\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"
Sep 15 11:26:06.874246 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.874100    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3fcdb73-b50f-41bb-a829-551da06facff-tmp\") pod \"klusterlet-addon-workmgr-59f6487989-8xxx8\" (UID: \"a3fcdb73-b50f-41bb-a829-551da06facff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f6487989-8xxx8"
Sep 15 11:26:06.874570 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.874535    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b085c684-f385-4e33-b780-492f1aac71b8-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-p6cx4\" (UID: \"b085c684-f385-4e33-b780-492f1aac71b8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6cx4"
Sep 15 11:26:06.876155 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.876028    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b085c684-f385-4e33-b780-492f1aac71b8-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-p6cx4\" (UID: \"b085c684-f385-4e33-b780-492f1aac71b8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6cx4"
Sep 15 11:26:06.876155 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.876092    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/90992df9-ddc5-413d-93c6-5cb6c9cd5038-hub\") pod \"cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5\" (UID: \"90992df9-ddc5-413d-93c6-5cb6c9cd5038\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"
Sep 15 11:26:06.876346 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.876305    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/90992df9-ddc5-413d-93c6-5cb6c9cd5038-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5\" (UID: \"90992df9-ddc5-413d-93c6-5cb6c9cd5038\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"
Sep 15 11:26:06.876701 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.876678    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/90992df9-ddc5-413d-93c6-5cb6c9cd5038-ca\") pod \"cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5\" (UID: \"90992df9-ddc5-413d-93c6-5cb6c9cd5038\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"
Sep 15 11:26:06.876793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.876684    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a3fcdb73-b50f-41bb-a829-551da06facff-klusterlet-config\") pod \"klusterlet-addon-workmgr-59f6487989-8xxx8\" (UID: \"a3fcdb73-b50f-41bb-a829-551da06facff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f6487989-8xxx8"
Sep 15 11:26:06.876793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.876722    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7afafd1c-e60e-447f-8d8a-5441a20af1ec-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-57bfccd9fd-ppnrc\" (UID: \"7afafd1c-e60e-447f-8d8a-5441a20af1ec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bfccd9fd-ppnrc"
Sep 15 11:26:06.877192 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.877174    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90992df9-ddc5-413d-93c6-5cb6c9cd5038-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5\" (UID: \"90992df9-ddc5-413d-93c6-5cb6c9cd5038\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"
Sep 15 11:26:06.882369 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.882345    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg2pr\" (UniqueName: \"kubernetes.io/projected/a3fcdb73-b50f-41bb-a829-551da06facff-kube-api-access-kg2pr\") pod \"klusterlet-addon-workmgr-59f6487989-8xxx8\" (UID: \"a3fcdb73-b50f-41bb-a829-551da06facff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f6487989-8xxx8"
Sep 15 11:26:06.883458 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.883430    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkkwh\" (UniqueName: \"kubernetes.io/projected/90992df9-ddc5-413d-93c6-5cb6c9cd5038-kube-api-access-tkkwh\") pod \"cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5\" (UID: \"90992df9-ddc5-413d-93c6-5cb6c9cd5038\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"
Sep 15 11:26:06.883979 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.883961    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnh9w\" (UniqueName: \"kubernetes.io/projected/7afafd1c-e60e-447f-8d8a-5441a20af1ec-kube-api-access-qnh9w\") pod \"managed-serviceaccount-addon-agent-57bfccd9fd-ppnrc\" (UID: \"7afafd1c-e60e-447f-8d8a-5441a20af1ec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bfccd9fd-ppnrc"
Sep 15 11:26:06.974832 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.974747    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-trusted-ca\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.974832 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.974809    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-registry-certificates\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.975115 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.974843    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2kdh\" (UniqueName: \"kubernetes.io/projected/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-kube-api-access-q2kdh\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.975115 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.974876    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-ca-trust-extracted\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.975115 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.974894    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-registry-tls\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.975115 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.974956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-image-registry-private-configuration\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.975115 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.975012    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-installation-pull-secrets\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.975115 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.975036    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-bound-sa-token\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.975679 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.975648    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-ca-trust-extracted\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.975802 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.975732    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-trusted-ca\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.975802 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.975761    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-registry-certificates\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.976929 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.976913    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6cx4"
Sep 15 11:26:06.977582 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.977541    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-installation-pull-secrets\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.977891 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.977872    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-registry-tls\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.978019 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.977996    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-image-registry-private-configuration\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.986154 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.986126    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2kdh\" (UniqueName: \"kubernetes.io/projected/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-kube-api-access-q2kdh\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.986359 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.986341    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0fb1ee35-a207-42fa-b165-dfa8c7e93a86-bound-sa-token\") pod \"image-registry-759494458-fl68h\" (UID: \"0fb1ee35-a207-42fa-b165-dfa8c7e93a86\") " pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:06.988259 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:06.988237    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f6487989-8xxx8"
Sep 15 11:26:07.002359 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.002339    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bfccd9fd-ppnrc"
Sep 15 11:26:07.008343 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.008322    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"
Sep 15 11:26:07.085456 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.085312    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:07.254994 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.254923    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-p6cx4"]
Sep 15 11:26:07.258132 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:26:07.258106    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb085c684_f385_4e33_b780_492f1aac71b8.slice/crio-ddd8a27ff82b4ff8de402b40426d4bc0f1454f7d3869c23771a23bdfba248acb WatchSource:0}: Error finding container ddd8a27ff82b4ff8de402b40426d4bc0f1454f7d3869c23771a23bdfba248acb: Status 404 returned error can't find the container with id ddd8a27ff82b4ff8de402b40426d4bc0f1454f7d3869c23771a23bdfba248acb
Sep 15 11:26:07.265018 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.264994    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6cx4" event={"ID":"b085c684-f385-4e33-b780-492f1aac71b8","Type":"ContainerStarted","Data":"ddd8a27ff82b4ff8de402b40426d4bc0f1454f7d3869c23771a23bdfba248acb"}
Sep 15 11:26:07.276680 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.276656    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f6487989-8xxx8"]
Sep 15 11:26:07.279514 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:26:07.279494    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3fcdb73_b50f_41bb_a829_551da06facff.slice/crio-1f6246fc3d3f4c237d8b43f74ae18aaa212b68aefba7f52b4e68b99e57d756cf WatchSource:0}: Error finding container 1f6246fc3d3f4c237d8b43f74ae18aaa212b68aefba7f52b4e68b99e57d756cf: Status 404 returned error can't find the container with id 1f6246fc3d3f4c237d8b43f74ae18aaa212b68aefba7f52b4e68b99e57d756cf
Sep 15 11:26:07.295417 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.295397    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bfccd9fd-ppnrc"]
Sep 15 11:26:07.301401 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:26:07.301365    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7afafd1c_e60e_447f_8d8a_5441a20af1ec.slice/crio-f8a5fb8d4753554f3a7f8193f0529eda5a49b5d30c1a069615dc25c69d09e26c WatchSource:0}: Error finding container f8a5fb8d4753554f3a7f8193f0529eda5a49b5d30c1a069615dc25c69d09e26c: Status 404 returned error can't find the container with id f8a5fb8d4753554f3a7f8193f0529eda5a49b5d30c1a069615dc25c69d09e26c
Sep 15 11:26:07.311304 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.311269    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5"]
Sep 15 11:26:07.314265 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:26:07.314240    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90992df9_ddc5_413d_93c6_5cb6c9cd5038.slice/crio-322521b276a288bc0c3fcc17526c82cc40d1a919ff1a9b61323a48eba0d9bd9d WatchSource:0}: Error finding container 322521b276a288bc0c3fcc17526c82cc40d1a919ff1a9b61323a48eba0d9bd9d: Status 404 returned error can't find the container with id 322521b276a288bc0c3fcc17526c82cc40d1a919ff1a9b61323a48eba0d9bd9d
Sep 15 11:26:07.360709 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.360676    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-759494458-fl68h"]
Sep 15 11:26:07.364043 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:26:07.364019    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb1ee35_a207_42fa_b165_dfa8c7e93a86.slice/crio-20af3d5e350ebec957fe0bb5d0b3397c956d0852deba6aa4670b2b1919019a64 WatchSource:0}: Error finding container 20af3d5e350ebec957fe0bb5d0b3397c956d0852deba6aa4670b2b1919019a64: Status 404 returned error can't find the container with id 20af3d5e350ebec957fe0bb5d0b3397c956d0852deba6aa4670b2b1919019a64
Sep 15 11:26:07.580557 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.580462    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret\") pod \"global-pull-secret-syncer-7j9vh\" (UID: \"85b6a362-7f90-4e54-89d2-2c21ef687efb\") " pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:26:07.580557 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.580522    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs\") pod \"network-metrics-daemon-cc7mk\" (UID: \"355f2000-c95f-4d71-b450-03ce72b5ea62\") " pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:26:07.580557 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.580555    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6gd\" (UniqueName: \"kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd\") pod \"network-check-target-4gbpc\" (UID: \"a0d6a68c-1fa6-4f31-b6f8-3a1a38988836\") " pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:26:07.582999 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.582975    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 11:26:07.583167 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.582975    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 11:26:07.583255 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.583168    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q6gd\" (UniqueName: \"kubernetes.io/projected/a0d6a68c-1fa6-4f31-b6f8-3a1a38988836-kube-api-access-6q6gd\") pod \"network-check-target-4gbpc\" (UID: \"a0d6a68c-1fa6-4f31-b6f8-3a1a38988836\") " pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:26:07.592923 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.592901    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355f2000-c95f-4d71-b450-03ce72b5ea62-metrics-certs\") pod \"network-metrics-daemon-cc7mk\" (UID: \"355f2000-c95f-4d71-b450-03ce72b5ea62\") " pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:26:07.593073 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.592926    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85b6a362-7f90-4e54-89d2-2c21ef687efb-original-pull-secret\") pod \"global-pull-secret-syncer-7j9vh\" (UID: \"85b6a362-7f90-4e54-89d2-2c21ef687efb\") " pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:26:07.679455 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.679409    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n59hg\""
Sep 15 11:26:07.683233 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.683204    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7j9vh"
Sep 15 11:26:07.687662 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.687643    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:26:07.689677 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.689657    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4fvsz\""
Sep 15 11:26:07.698430 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.698407    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cc7mk"
Sep 15 11:26:07.964641 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.964603    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7j9vh"]
Sep 15 11:26:07.966939 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.966844    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4gbpc"]
Sep 15 11:26:07.969358 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:26:07.969335    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b6a362_7f90_4e54_89d2_2c21ef687efb.slice/crio-01452c1513983edf0623033641b518ea16d6112744e508fb0a4d9dd25a8937dc WatchSource:0}: Error finding container 01452c1513983edf0623033641b518ea16d6112744e508fb0a4d9dd25a8937dc: Status 404 returned error can't find the container with id 01452c1513983edf0623033641b518ea16d6112744e508fb0a4d9dd25a8937dc
Sep 15 11:26:07.970743 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:26:07.970715    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d6a68c_1fa6_4f31_b6f8_3a1a38988836.slice/crio-0aec916554cb395072b2ad81c3160157987839604103e532e171d4fabdc9baa2 WatchSource:0}: Error finding container 0aec916554cb395072b2ad81c3160157987839604103e532e171d4fabdc9baa2: Status 404 returned error can't find the container with id 0aec916554cb395072b2ad81c3160157987839604103e532e171d4fabdc9baa2
Sep 15 11:26:07.990736 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:07.990645    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cc7mk"]
Sep 15 11:26:07.999120 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:26:07.999093    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355f2000_c95f_4d71_b450_03ce72b5ea62.slice/crio-8ea5a87d4da53cdeabb51b021cd92ef4f84b33f8e0308f3cf40872b5ee39f446 WatchSource:0}: Error finding container 8ea5a87d4da53cdeabb51b021cd92ef4f84b33f8e0308f3cf40872b5ee39f446: Status 404 returned error can't find the container with id 8ea5a87d4da53cdeabb51b021cd92ef4f84b33f8e0308f3cf40872b5ee39f446
Sep 15 11:26:08.270000 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:08.269915    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f6487989-8xxx8" event={"ID":"a3fcdb73-b50f-41bb-a829-551da06facff","Type":"ContainerStarted","Data":"1f6246fc3d3f4c237d8b43f74ae18aaa212b68aefba7f52b4e68b99e57d756cf"}
Sep 15 11:26:08.271764 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:08.271737    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-759494458-fl68h" event={"ID":"0fb1ee35-a207-42fa-b165-dfa8c7e93a86","Type":"ContainerStarted","Data":"0602f89a8e8e5ca78cb2dd60e8a07065d3e2b2d0bc0ae647b31f1d458d3407c5"}
Sep 15 11:26:08.271886 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:08.271784    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-759494458-fl68h" event={"ID":"0fb1ee35-a207-42fa-b165-dfa8c7e93a86","Type":"ContainerStarted","Data":"20af3d5e350ebec957fe0bb5d0b3397c956d0852deba6aa4670b2b1919019a64"}
Sep 15 11:26:08.271985 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:08.271946    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:08.273107 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:08.273082    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bfccd9fd-ppnrc" event={"ID":"7afafd1c-e60e-447f-8d8a-5441a20af1ec","Type":"ContainerStarted","Data":"f8a5fb8d4753554f3a7f8193f0529eda5a49b5d30c1a069615dc25c69d09e26c"}
Sep 15 11:26:08.274770 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:08.274748    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4gbpc" event={"ID":"a0d6a68c-1fa6-4f31-b6f8-3a1a38988836","Type":"ContainerStarted","Data":"761141f512b8505d0a53adde6ebac172877d9b4a69a25dcc08685cd87d4b0fc8"}
Sep 15 11:26:08.274856 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:08.274792    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4gbpc" event={"ID":"a0d6a68c-1fa6-4f31-b6f8-3a1a38988836","Type":"ContainerStarted","Data":"0aec916554cb395072b2ad81c3160157987839604103e532e171d4fabdc9baa2"}
Sep 15 11:26:08.274919 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:08.274862    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:26:08.276109 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:08.276086    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7j9vh" event={"ID":"85b6a362-7f90-4e54-89d2-2c21ef687efb","Type":"ContainerStarted","Data":"01452c1513983edf0623033641b518ea16d6112744e508fb0a4d9dd25a8937dc"}
Sep 15 11:26:08.277375 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:08.277351    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cc7mk" event={"ID":"355f2000-c95f-4d71-b450-03ce72b5ea62","Type":"ContainerStarted","Data":"8ea5a87d4da53cdeabb51b021cd92ef4f84b33f8e0308f3cf40872b5ee39f446"}
Sep 15 11:26:08.278643 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:08.278621    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5" event={"ID":"90992df9-ddc5-413d-93c6-5cb6c9cd5038","Type":"ContainerStarted","Data":"322521b276a288bc0c3fcc17526c82cc40d1a919ff1a9b61323a48eba0d9bd9d"}
Sep 15 11:26:08.292432 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:08.292388    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-759494458-fl68h" podStartSLOduration=2.292373103 podStartE2EDuration="2.292373103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:08.291079545 +0000 UTC m=+66.953574754" watchObservedRunningTime="2026-09-15 11:26:08.292373103 +0000 UTC m=+66.954868287"
Sep 15 11:26:08.306546 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:08.306490    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4gbpc" podStartSLOduration=66.306475261 podStartE2EDuration="1m6.306475261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:08.305258256 +0000 UTC m=+66.967753461" watchObservedRunningTime="2026-09-15 11:26:08.306475261 +0000 UTC m=+66.968970442"
Sep 15 11:26:09.294465 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:09.294424    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6cx4" event={"ID":"b085c684-f385-4e33-b780-492f1aac71b8","Type":"ContainerStarted","Data":"4ce4433b99aaa00a06a68dcf6a82efccd438279355a7fddf235c963ea20d291c"}
Sep 15 11:26:09.312628 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:09.312471    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-p6cx4" podStartSLOduration=2.139565041 podStartE2EDuration="3.312448134s" totalImagesPullingTime="1.172883093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:09.310371883 +0000 UTC m=+67.972867065" watchObservedRunningTime="2026-09-15 11:26:09.312448134 +0000 UTC m=+67.974943317"
Sep 15 11:26:10.315796 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:10.315754    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cc7mk" event={"ID":"355f2000-c95f-4d71-b450-03ce72b5ea62","Type":"ContainerStarted","Data":"b8ecd311b89a067853f4f3cdd62ab9f24df14e59ec12431a4ab6151089ae4f87"}
Sep 15 11:26:10.316285 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:10.315817    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cc7mk" event={"ID":"355f2000-c95f-4d71-b450-03ce72b5ea62","Type":"ContainerStarted","Data":"ac4232dc9a5d79bfa8d7504c5df0be0bc5056c8c22e7c9f5df0f0fc8f5b8f313"}
Sep 15 11:26:10.336701 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:10.336646    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cc7mk" podStartSLOduration=66.467692257 podStartE2EDuration="1m8.336623945s" totalImagesPullingTime="1.868931688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:10.334879198 +0000 UTC m=+68.997374382" watchObservedRunningTime="2026-09-15 11:26:10.336623945 +0000 UTC m=+68.999119127"
Sep 15 11:26:16.706891 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:16.706861    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:26:17.347265 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:17.347223    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5" event={"ID":"90992df9-ddc5-413d-93c6-5cb6c9cd5038","Type":"ContainerStarted","Data":"b0b5f4a6aa4fc45d7890eeecc192c855f4dd927a81ab9335319beebc9727ce8e"}
Sep 15 11:26:17.347469 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:17.347275    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5" event={"ID":"90992df9-ddc5-413d-93c6-5cb6c9cd5038","Type":"ContainerStarted","Data":"cde52dde87c3cb26a4d9b37ca7dce2f41fd07138a9ccc26e904a3f6a9929bd82"}
Sep 15 11:26:17.347469 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:17.347294    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5" event={"ID":"90992df9-ddc5-413d-93c6-5cb6c9cd5038","Type":"ContainerStarted","Data":"9e14f6b2049a7130d2ded68f0467356a6cb2d7b2dd957f24e24d8cd6fa4f72bf"}
Sep 15 11:26:17.349015 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:17.348981    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f6487989-8xxx8" event={"ID":"a3fcdb73-b50f-41bb-a829-551da06facff","Type":"ContainerStarted","Data":"a2b3caabb4a4dd77c2cc49463a657a8fa39c1f8014c1c38f02ba24a00ef4a47d"}
Sep 15 11:26:17.349183 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:17.349042    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f6487989-8xxx8"
Sep 15 11:26:17.351090 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:17.351044    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bfccd9fd-ppnrc" event={"ID":"7afafd1c-e60e-447f-8d8a-5441a20af1ec","Type":"ContainerStarted","Data":"875b526c40805f156cda2368658e3e61b0a6e979730445ef431ce406ef61882f"}
Sep 15 11:26:17.351269 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:17.351252    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f6487989-8xxx8"
Sep 15 11:26:17.352793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:17.352767    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7j9vh" event={"ID":"85b6a362-7f90-4e54-89d2-2c21ef687efb","Type":"ContainerStarted","Data":"9e2192e47a1a2fe549140b069ac07430aece49103256d02e959f131324a18cea"}
Sep 15 11:26:17.366695 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:17.366646    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cdbb9d56b-xx7z5" podStartSLOduration=1.92628244 podStartE2EDuration="11.366631906s" totalImagesPullingTime="9.440349466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:17.36615111 +0000 UTC m=+76.028646294" watchObservedRunningTime="2026-09-15 11:26:17.366631906 +0000 UTC m=+76.029127078"
Sep 15 11:26:17.383450 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:17.383406    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f6487989-8xxx8" podStartSLOduration=1.923962302 podStartE2EDuration="11.383390753s" totalImagesPullingTime="9.459428451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:17.382574651 +0000 UTC m=+76.045069855" watchObservedRunningTime="2026-09-15 11:26:17.383390753 +0000 UTC m=+76.045885931"
Sep 15 11:26:17.407027 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:17.406977    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bfccd9fd-ppnrc" podStartSLOduration=1.955648247 podStartE2EDuration="11.406961398s" totalImagesPullingTime="9.451313151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:17.406442848 +0000 UTC m=+76.068938030" watchObservedRunningTime="2026-09-15 11:26:17.406961398 +0000 UTC m=+76.069456942"
Sep 15 11:26:17.425369 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:17.425324    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7j9vh" podStartSLOduration=66.639371338 podStartE2EDuration="1m15.425310331s" totalImagesPullingTime="8.785938993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:17.424437406 +0000 UTC m=+76.086932608" watchObservedRunningTime="2026-09-15 11:26:17.425310331 +0000 UTC m=+76.087805521"
Sep 15 11:26:23.373865 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.373827    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qtn7k"]
Sep 15 11:26:23.376459 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.376437    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.379029 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.379003    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 11:26:23.379029 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.379017    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 11:26:23.379372 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.379303    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k5ffd\""
Sep 15 11:26:23.379372 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.379337    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 11:26:23.379557 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.379393    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 11:26:23.379922 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.379684    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 11:26:23.379922 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.379746    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 11:26:23.516821 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.516781    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fbb571d-8b56-4afb-ba33-04283dc19f69-metrics-client-ca\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.516821 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.516831    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fbb571d-8b56-4afb-ba33-04283dc19f69-sys\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.517035 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.516907    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9fbb571d-8b56-4afb-ba33-04283dc19f69-node-exporter-accelerators-collector-config\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.517035 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.516973    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fbb571d-8b56-4afb-ba33-04283dc19f69-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.517035 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.517003    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzbk\" (UniqueName: \"kubernetes.io/projected/9fbb571d-8b56-4afb-ba33-04283dc19f69-kube-api-access-6pzbk\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.517157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.517034    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fbb571d-8b56-4afb-ba33-04283dc19f69-node-exporter-tls\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.517157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.517134    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9fbb571d-8b56-4afb-ba33-04283dc19f69-root\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.517229 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.517162    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9fbb571d-8b56-4afb-ba33-04283dc19f69-node-exporter-textfile\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.517229 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.517199    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9fbb571d-8b56-4afb-ba33-04283dc19f69-node-exporter-wtmp\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.618192 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.618147    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9fbb571d-8b56-4afb-ba33-04283dc19f69-node-exporter-wtmp\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.618390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.618202    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fbb571d-8b56-4afb-ba33-04283dc19f69-metrics-client-ca\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.618390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.618227    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fbb571d-8b56-4afb-ba33-04283dc19f69-sys\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.618390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.618249    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9fbb571d-8b56-4afb-ba33-04283dc19f69-node-exporter-accelerators-collector-config\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.618390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.618291    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fbb571d-8b56-4afb-ba33-04283dc19f69-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.618390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.618298    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fbb571d-8b56-4afb-ba33-04283dc19f69-sys\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.618390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.618336    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9fbb571d-8b56-4afb-ba33-04283dc19f69-node-exporter-wtmp\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.618390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.618314    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzbk\" (UniqueName: \"kubernetes.io/projected/9fbb571d-8b56-4afb-ba33-04283dc19f69-kube-api-access-6pzbk\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.618390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.618389    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fbb571d-8b56-4afb-ba33-04283dc19f69-node-exporter-tls\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.618782 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.618464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9fbb571d-8b56-4afb-ba33-04283dc19f69-root\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.618782 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.618494    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9fbb571d-8b56-4afb-ba33-04283dc19f69-node-exporter-textfile\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.618782 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:26:23.618669    2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 15 11:26:23.618782 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.618674    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9fbb571d-8b56-4afb-ba33-04283dc19f69-root\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.618782 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:26:23.618737    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fbb571d-8b56-4afb-ba33-04283dc19f69-node-exporter-tls podName:9fbb571d-8b56-4afb-ba33-04283dc19f69 nodeName:}" failed. No retries permitted until 2026-09-15 11:26:24.118716356 +0000 UTC m=+82.781211516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9fbb571d-8b56-4afb-ba33-04283dc19f69-node-exporter-tls") pod "node-exporter-qtn7k" (UID: "9fbb571d-8b56-4afb-ba33-04283dc19f69") : secret "node-exporter-tls" not found
Sep 15 11:26:23.619035 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.618905    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9fbb571d-8b56-4afb-ba33-04283dc19f69-node-exporter-accelerators-collector-config\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.619035 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.618948    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fbb571d-8b56-4afb-ba33-04283dc19f69-metrics-client-ca\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.619035 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.618956    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9fbb571d-8b56-4afb-ba33-04283dc19f69-node-exporter-textfile\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.620853 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.620830    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fbb571d-8b56-4afb-ba33-04283dc19f69-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:23.626032 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:23.625983    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzbk\" (UniqueName: \"kubernetes.io/projected/9fbb571d-8b56-4afb-ba33-04283dc19f69-kube-api-access-6pzbk\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:24.124168 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:24.124126    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fbb571d-8b56-4afb-ba33-04283dc19f69-node-exporter-tls\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:24.126405 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:24.126383    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fbb571d-8b56-4afb-ba33-04283dc19f69-node-exporter-tls\") pod \"node-exporter-qtn7k\" (UID: \"9fbb571d-8b56-4afb-ba33-04283dc19f69\") " pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:24.287375 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:24.287332    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qtn7k"
Sep 15 11:26:24.300108 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:26:24.300072    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbb571d_8b56_4afb_ba33_04283dc19f69.slice/crio-4e86f507a9234ff8939239f343347142c85da2e9b83a6ad7e2e20ad627b41dfc WatchSource:0}: Error finding container 4e86f507a9234ff8939239f343347142c85da2e9b83a6ad7e2e20ad627b41dfc: Status 404 returned error can't find the container with id 4e86f507a9234ff8939239f343347142c85da2e9b83a6ad7e2e20ad627b41dfc
Sep 15 11:26:24.378784 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:24.378697    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qtn7k" event={"ID":"9fbb571d-8b56-4afb-ba33-04283dc19f69","Type":"ContainerStarted","Data":"4e86f507a9234ff8939239f343347142c85da2e9b83a6ad7e2e20ad627b41dfc"}
Sep 15 11:26:27.392216 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:27.392176    2562 generic.go:356] "Generic (PLEG): container finished" podID="9fbb571d-8b56-4afb-ba33-04283dc19f69" containerID="e9852875c419032b258aee5bcb242b241da1535fd85e11a2f9873de56e273938" exitCode=0
Sep 15 11:26:27.392705 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:27.392275    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qtn7k" event={"ID":"9fbb571d-8b56-4afb-ba33-04283dc19f69","Type":"ContainerDied","Data":"e9852875c419032b258aee5bcb242b241da1535fd85e11a2f9873de56e273938"}
Sep 15 11:26:28.398616 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:28.398570    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qtn7k" event={"ID":"9fbb571d-8b56-4afb-ba33-04283dc19f69","Type":"ContainerStarted","Data":"699d4f23f4ec3dbadbf917211b0416fafb2a3bc047073b2af9bb3f37ce3da0cd"}
Sep 15 11:26:28.398997 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:28.398641    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qtn7k" event={"ID":"9fbb571d-8b56-4afb-ba33-04283dc19f69","Type":"ContainerStarted","Data":"3beca9fe71b86f1f582dc90e0e8ae6951c0cc1c5a3fab969ae25f22cff7c7a6b"}
Sep 15 11:26:28.420752 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:28.420688    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qtn7k" podStartSLOduration=2.814653975 podStartE2EDuration="5.42066898s" totalImagesPullingTime="2.5457094s" totalInitContainerRuntime="60.305605ms" isStatefulPod=true podCreationTimestamp="2026-09-15 11:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:28.420390559 +0000 UTC m=+87.082885739" watchObservedRunningTime="2026-09-15 11:26:28.42066898 +0000 UTC m=+87.083164163"
Sep 15 11:26:29.299759 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:29.299730    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-759494458-fl68h"
Sep 15 11:26:31.721259 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:31.721192    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw" podUID="f7547112-c429-4178-a48e-f47e6ef7576c" containerName="registry" containerID="cri-o://608404d21c4c6a71280256ec4bbc78f0c0e1ea634b49acf611343896d661913c" gracePeriod=30
Sep 15 11:26:31.962129 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:31.962101    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:26:32.088748 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.088715    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-bound-sa-token\") pod \"f7547112-c429-4178-a48e-f47e6ef7576c\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") "
Sep 15 11:26:32.088932 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.088757    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7547112-c429-4178-a48e-f47e6ef7576c-installation-pull-secrets\") pod \"f7547112-c429-4178-a48e-f47e6ef7576c\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") "
Sep 15 11:26:32.088932 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.088785    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7547112-c429-4178-a48e-f47e6ef7576c-registry-certificates\") pod \"f7547112-c429-4178-a48e-f47e6ef7576c\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") "
Sep 15 11:26:32.088932 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.088817    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls\") pod \"f7547112-c429-4178-a48e-f47e6ef7576c\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") "
Sep 15 11:26:32.088932 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.088861    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7547112-c429-4178-a48e-f47e6ef7576c-ca-trust-extracted\") pod \"f7547112-c429-4178-a48e-f47e6ef7576c\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") "
Sep 15 11:26:32.089174 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.089021    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7547112-c429-4178-a48e-f47e6ef7576c-trusted-ca\") pod \"f7547112-c429-4178-a48e-f47e6ef7576c\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") "
Sep 15 11:26:32.089174 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.089096    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7547112-c429-4178-a48e-f47e6ef7576c-image-registry-private-configuration\") pod \"f7547112-c429-4178-a48e-f47e6ef7576c\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") "
Sep 15 11:26:32.089174 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.089146    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thfdf\" (UniqueName: \"kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-kube-api-access-thfdf\") pod \"f7547112-c429-4178-a48e-f47e6ef7576c\" (UID: \"f7547112-c429-4178-a48e-f47e6ef7576c\") "
Sep 15 11:26:32.089386 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.089358    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7547112-c429-4178-a48e-f47e6ef7576c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f7547112-c429-4178-a48e-f47e6ef7576c" (UID: "f7547112-c429-4178-a48e-f47e6ef7576c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:26:32.089762 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.089732    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7547112-c429-4178-a48e-f47e6ef7576c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f7547112-c429-4178-a48e-f47e6ef7576c" (UID: "f7547112-c429-4178-a48e-f47e6ef7576c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:26:32.091251 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.091226    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7547112-c429-4178-a48e-f47e6ef7576c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f7547112-c429-4178-a48e-f47e6ef7576c" (UID: "f7547112-c429-4178-a48e-f47e6ef7576c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:26:32.091394 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.091339    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f7547112-c429-4178-a48e-f47e6ef7576c" (UID: "f7547112-c429-4178-a48e-f47e6ef7576c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:26:32.091552 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.091521    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f7547112-c429-4178-a48e-f47e6ef7576c" (UID: "f7547112-c429-4178-a48e-f47e6ef7576c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:26:32.091662 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.091636    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7547112-c429-4178-a48e-f47e6ef7576c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f7547112-c429-4178-a48e-f47e6ef7576c" (UID: "f7547112-c429-4178-a48e-f47e6ef7576c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:26:32.091885 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.091864    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-kube-api-access-thfdf" (OuterVolumeSpecName: "kube-api-access-thfdf") pod "f7547112-c429-4178-a48e-f47e6ef7576c" (UID: "f7547112-c429-4178-a48e-f47e6ef7576c"). InnerVolumeSpecName "kube-api-access-thfdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:26:32.097324 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.097299    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7547112-c429-4178-a48e-f47e6ef7576c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f7547112-c429-4178-a48e-f47e6ef7576c" (UID: "f7547112-c429-4178-a48e-f47e6ef7576c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:26:32.190437 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.190396    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7547112-c429-4178-a48e-f47e6ef7576c-trusted-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:26:32.190617 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.190434    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7547112-c429-4178-a48e-f47e6ef7576c-image-registry-private-configuration\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:26:32.190617 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.190461    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thfdf\" (UniqueName: \"kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-kube-api-access-thfdf\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:26:32.190617 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.190474    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-bound-sa-token\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:26:32.190617 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.190485    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7547112-c429-4178-a48e-f47e6ef7576c-installation-pull-secrets\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:26:32.190617 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.190496    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7547112-c429-4178-a48e-f47e6ef7576c-registry-certificates\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:26:32.190617 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.190507    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7547112-c429-4178-a48e-f47e6ef7576c-registry-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:26:32.190617 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.190518    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7547112-c429-4178-a48e-f47e6ef7576c-ca-trust-extracted\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:26:32.412777 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.412694    2562 generic.go:356] "Generic (PLEG): container finished" podID="f7547112-c429-4178-a48e-f47e6ef7576c" containerID="608404d21c4c6a71280256ec4bbc78f0c0e1ea634b49acf611343896d661913c" exitCode=0
Sep 15 11:26:32.412777 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.412742    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw"
Sep 15 11:26:32.412777 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.412754    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw" event={"ID":"f7547112-c429-4178-a48e-f47e6ef7576c","Type":"ContainerDied","Data":"608404d21c4c6a71280256ec4bbc78f0c0e1ea634b49acf611343896d661913c"}
Sep 15 11:26:32.413001 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.412794    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86d58d4fdf-q72nw" event={"ID":"f7547112-c429-4178-a48e-f47e6ef7576c","Type":"ContainerDied","Data":"3771ce37aa4eed6ce7403a5a03d431d49fcbb139b6198a33aa6adcecbdd0c95d"}
Sep 15 11:26:32.413001 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.412816    2562 scope.go:117] "RemoveContainer" containerID="608404d21c4c6a71280256ec4bbc78f0c0e1ea634b49acf611343896d661913c"
Sep 15 11:26:32.421738 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.421721    2562 scope.go:117] "RemoveContainer" containerID="608404d21c4c6a71280256ec4bbc78f0c0e1ea634b49acf611343896d661913c"
Sep 15 11:26:32.422020 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:26:32.422005    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"608404d21c4c6a71280256ec4bbc78f0c0e1ea634b49acf611343896d661913c\": container with ID starting with 608404d21c4c6a71280256ec4bbc78f0c0e1ea634b49acf611343896d661913c not found: ID does not exist" containerID="608404d21c4c6a71280256ec4bbc78f0c0e1ea634b49acf611343896d661913c"
Sep 15 11:26:32.422082 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.422028    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608404d21c4c6a71280256ec4bbc78f0c0e1ea634b49acf611343896d661913c"} err="failed to get container status \"608404d21c4c6a71280256ec4bbc78f0c0e1ea634b49acf611343896d661913c\": rpc error: code = NotFound desc = could not find container \"608404d21c4c6a71280256ec4bbc78f0c0e1ea634b49acf611343896d661913c\": container with ID starting with 608404d21c4c6a71280256ec4bbc78f0c0e1ea634b49acf611343896d661913c not found: ID does not exist"
Sep 15 11:26:32.432506 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.432478    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86d58d4fdf-q72nw"]
Sep 15 11:26:32.436574 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:32.436551    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-86d58d4fdf-q72nw"]
Sep 15 11:26:33.875980 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:33.875940    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7547112-c429-4178-a48e-f47e6ef7576c" path="/var/lib/kubelet/pods/f7547112-c429-4178-a48e-f47e6ef7576c/volumes"
Sep 15 11:26:39.298066 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:26:39.298021    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4gbpc"
Sep 15 11:30:01.766319 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:01.766289    2562 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 11:30:27.807812 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.807776    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"]
Sep 15 11:30:27.809180 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.808139    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7547112-c429-4178-a48e-f47e6ef7576c" containerName="registry"
Sep 15 11:30:27.809180 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.808154    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7547112-c429-4178-a48e-f47e6ef7576c" containerName="registry"
Sep 15 11:30:27.809180 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.808212    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7547112-c429-4178-a48e-f47e6ef7576c" containerName="registry"
Sep 15 11:30:27.810028 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.810008    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"
Sep 15 11:30:27.813476 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.813451    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 11:30:27.813743 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.813723    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 11:30:27.813743 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.813742    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 11:30:27.813894 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.813787    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vtkfp\""
Sep 15 11:30:27.814155 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.814141    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 11:30:27.831646 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.831617    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"]
Sep 15 11:30:27.886130 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.886096    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb818920-9a4c-4a0e-8899-7d62c5c3601d-tmp\") pod \"opendatahub-operator-controller-manager-b96c7c888-d4f46\" (UID: \"cb818920-9a4c-4a0e-8899-7d62c5c3601d\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"
Sep 15 11:30:27.886299 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.886153    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb818920-9a4c-4a0e-8899-7d62c5c3601d-webhook-cert\") pod \"opendatahub-operator-controller-manager-b96c7c888-d4f46\" (UID: \"cb818920-9a4c-4a0e-8899-7d62c5c3601d\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"
Sep 15 11:30:27.886299 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.886175    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xnjz\" (UniqueName: \"kubernetes.io/projected/cb818920-9a4c-4a0e-8899-7d62c5c3601d-kube-api-access-8xnjz\") pod \"opendatahub-operator-controller-manager-b96c7c888-d4f46\" (UID: \"cb818920-9a4c-4a0e-8899-7d62c5c3601d\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"
Sep 15 11:30:27.886299 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.886222    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cb818920-9a4c-4a0e-8899-7d62c5c3601d-manifests\") pod \"opendatahub-operator-controller-manager-b96c7c888-d4f46\" (UID: \"cb818920-9a4c-4a0e-8899-7d62c5c3601d\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"
Sep 15 11:30:27.886299 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.886242    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cb818920-9a4c-4a0e-8899-7d62c5c3601d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b96c7c888-d4f46\" (UID: \"cb818920-9a4c-4a0e-8899-7d62c5c3601d\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"
Sep 15 11:30:27.987320 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.987274    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xnjz\" (UniqueName: \"kubernetes.io/projected/cb818920-9a4c-4a0e-8899-7d62c5c3601d-kube-api-access-8xnjz\") pod \"opendatahub-operator-controller-manager-b96c7c888-d4f46\" (UID: \"cb818920-9a4c-4a0e-8899-7d62c5c3601d\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"
Sep 15 11:30:27.987497 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.987334    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cb818920-9a4c-4a0e-8899-7d62c5c3601d-manifests\") pod \"opendatahub-operator-controller-manager-b96c7c888-d4f46\" (UID: \"cb818920-9a4c-4a0e-8899-7d62c5c3601d\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"
Sep 15 11:30:27.987497 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.987355    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cb818920-9a4c-4a0e-8899-7d62c5c3601d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b96c7c888-d4f46\" (UID: \"cb818920-9a4c-4a0e-8899-7d62c5c3601d\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"
Sep 15 11:30:27.987573 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.987519    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb818920-9a4c-4a0e-8899-7d62c5c3601d-tmp\") pod \"opendatahub-operator-controller-manager-b96c7c888-d4f46\" (UID: \"cb818920-9a4c-4a0e-8899-7d62c5c3601d\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"
Sep 15 11:30:27.987622 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.987605    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb818920-9a4c-4a0e-8899-7d62c5c3601d-webhook-cert\") pod \"opendatahub-operator-controller-manager-b96c7c888-d4f46\" (UID: \"cb818920-9a4c-4a0e-8899-7d62c5c3601d\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"
Sep 15 11:30:27.987799 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.987778    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cb818920-9a4c-4a0e-8899-7d62c5c3601d-manifests\") pod \"opendatahub-operator-controller-manager-b96c7c888-d4f46\" (UID: \"cb818920-9a4c-4a0e-8899-7d62c5c3601d\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"
Sep 15 11:30:27.989857 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.989829    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb818920-9a4c-4a0e-8899-7d62c5c3601d-tmp\") pod \"opendatahub-operator-controller-manager-b96c7c888-d4f46\" (UID: \"cb818920-9a4c-4a0e-8899-7d62c5c3601d\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"
Sep 15 11:30:27.989965 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.989905    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cb818920-9a4c-4a0e-8899-7d62c5c3601d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b96c7c888-d4f46\" (UID: \"cb818920-9a4c-4a0e-8899-7d62c5c3601d\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"
Sep 15 11:30:27.989965 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.989928    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb818920-9a4c-4a0e-8899-7d62c5c3601d-webhook-cert\") pod \"opendatahub-operator-controller-manager-b96c7c888-d4f46\" (UID: \"cb818920-9a4c-4a0e-8899-7d62c5c3601d\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"
Sep 15 11:30:27.999193 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:27.999165    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xnjz\" (UniqueName: \"kubernetes.io/projected/cb818920-9a4c-4a0e-8899-7d62c5c3601d-kube-api-access-8xnjz\") pod \"opendatahub-operator-controller-manager-b96c7c888-d4f46\" (UID: \"cb818920-9a4c-4a0e-8899-7d62c5c3601d\") " pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"
Sep 15 11:30:28.120905 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:28.120871    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"
Sep 15 11:30:28.402198 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:28.402175    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"]
Sep 15 11:30:28.404912 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:30:28.404874    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb818920_9a4c_4a0e_8899_7d62c5c3601d.slice/crio-72fa28b296b37c48e279f23fcde94c73f8e6df0f31256d79d9b19a727ce3e1a4 WatchSource:0}: Error finding container 72fa28b296b37c48e279f23fcde94c73f8e6df0f31256d79d9b19a727ce3e1a4: Status 404 returned error can't find the container with id 72fa28b296b37c48e279f23fcde94c73f8e6df0f31256d79d9b19a727ce3e1a4
Sep 15 11:30:28.406688 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:28.406670    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 11:30:29.222109 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:29.222050    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46" event={"ID":"cb818920-9a4c-4a0e-8899-7d62c5c3601d","Type":"ContainerStarted","Data":"72fa28b296b37c48e279f23fcde94c73f8e6df0f31256d79d9b19a727ce3e1a4"}
Sep 15 11:30:32.240954 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:32.240866    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46" event={"ID":"cb818920-9a4c-4a0e-8899-7d62c5c3601d","Type":"ContainerStarted","Data":"2cc997aa62d761a674b2bd47a31293f66930221ff5484de71cbd231c5faa03d9"}
Sep 15 11:30:33.246961 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:33.246870    2562 generic.go:356] "Generic (PLEG): container finished" podID="cb818920-9a4c-4a0e-8899-7d62c5c3601d" containerID="2cc997aa62d761a674b2bd47a31293f66930221ff5484de71cbd231c5faa03d9" exitCode=0
Sep 15 11:30:33.247348 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:33.246952    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46" event={"ID":"cb818920-9a4c-4a0e-8899-7d62c5c3601d","Type":"ContainerDied","Data":"2cc997aa62d761a674b2bd47a31293f66930221ff5484de71cbd231c5faa03d9"}
Sep 15 11:30:34.252632 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:34.252588    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46" event={"ID":"cb818920-9a4c-4a0e-8899-7d62c5c3601d","Type":"ContainerStarted","Data":"7d1c05b699b06c8b81de24f74498813b0d626c6f70c9a29b259282a4aa18f685"}
Sep 15 11:30:34.253023 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:34.252734    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"
Sep 15 11:30:34.293807 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:34.293748    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46" podStartSLOduration=2.446529137 podStartE2EDuration="7.293730849s" totalImagesPullingTime="3.807337743s" totalInitContainerRuntime="1.039863969s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:30:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:30:34.292565256 +0000 UTC m=+332.955060458" watchObservedRunningTime="2026-09-15 11:30:34.293730849 +0000 UTC m=+332.956226030"
Sep 15 11:30:45.259754 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:30:45.259719    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b96c7c888-d4f46"
Sep 15 11:31:04.288514 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.288426    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn"]
Sep 15 11:31:04.292304 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.292280    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn"
Sep 15 11:31:04.295380 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.295357    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 11:31:04.295990 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.295976    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 15 11:31:04.296190 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.296175    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-kc5kw\""
Sep 15 11:31:04.296248 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.296195    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 15 11:31:04.296313 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.296294    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 11:31:04.296350 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.296293    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 15 11:31:04.305985 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.305962    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn"]
Sep 15 11:31:04.383406 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.383367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsd9z\" (UniqueName: \"kubernetes.io/projected/3ed95d68-12dc-4162-a244-6113aeff38ba-kube-api-access-gsd9z\") pod \"lws-controller-manager-667dbb5fb5-ngctn\" (UID: \"3ed95d68-12dc-4162-a244-6113aeff38ba\") " pod="openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn"
Sep 15 11:31:04.383406 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.383411    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3ed95d68-12dc-4162-a244-6113aeff38ba-manager-config\") pod \"lws-controller-manager-667dbb5fb5-ngctn\" (UID: \"3ed95d68-12dc-4162-a244-6113aeff38ba\") " pod="openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn"
Sep 15 11:31:04.383622 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.383434    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ed95d68-12dc-4162-a244-6113aeff38ba-metrics-cert\") pod \"lws-controller-manager-667dbb5fb5-ngctn\" (UID: \"3ed95d68-12dc-4162-a244-6113aeff38ba\") " pod="openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn"
Sep 15 11:31:04.383622 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.383505    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ed95d68-12dc-4162-a244-6113aeff38ba-cert\") pod \"lws-controller-manager-667dbb5fb5-ngctn\" (UID: \"3ed95d68-12dc-4162-a244-6113aeff38ba\") " pod="openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn"
Sep 15 11:31:04.484731 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.484691    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ed95d68-12dc-4162-a244-6113aeff38ba-metrics-cert\") pod \"lws-controller-manager-667dbb5fb5-ngctn\" (UID: \"3ed95d68-12dc-4162-a244-6113aeff38ba\") " pod="openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn"
Sep 15 11:31:04.484731 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.484737    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ed95d68-12dc-4162-a244-6113aeff38ba-cert\") pod \"lws-controller-manager-667dbb5fb5-ngctn\" (UID: \"3ed95d68-12dc-4162-a244-6113aeff38ba\") " pod="openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn"
Sep 15 11:31:04.484993 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.484782    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsd9z\" (UniqueName: \"kubernetes.io/projected/3ed95d68-12dc-4162-a244-6113aeff38ba-kube-api-access-gsd9z\") pod \"lws-controller-manager-667dbb5fb5-ngctn\" (UID: \"3ed95d68-12dc-4162-a244-6113aeff38ba\") " pod="openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn"
Sep 15 11:31:04.484993 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.484806    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3ed95d68-12dc-4162-a244-6113aeff38ba-manager-config\") pod \"lws-controller-manager-667dbb5fb5-ngctn\" (UID: \"3ed95d68-12dc-4162-a244-6113aeff38ba\") " pod="openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn"
Sep 15 11:31:04.485449 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.485429    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3ed95d68-12dc-4162-a244-6113aeff38ba-manager-config\") pod \"lws-controller-manager-667dbb5fb5-ngctn\" (UID: \"3ed95d68-12dc-4162-a244-6113aeff38ba\") " pod="openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn"
Sep 15 11:31:04.487338 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.487313    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ed95d68-12dc-4162-a244-6113aeff38ba-metrics-cert\") pod \"lws-controller-manager-667dbb5fb5-ngctn\" (UID: \"3ed95d68-12dc-4162-a244-6113aeff38ba\") " pod="openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn"
Sep 15 11:31:04.487485 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.487465    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ed95d68-12dc-4162-a244-6113aeff38ba-cert\") pod \"lws-controller-manager-667dbb5fb5-ngctn\" (UID: \"3ed95d68-12dc-4162-a244-6113aeff38ba\") " pod="openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn"
Sep 15 11:31:04.493153 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.493127    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsd9z\" (UniqueName: \"kubernetes.io/projected/3ed95d68-12dc-4162-a244-6113aeff38ba-kube-api-access-gsd9z\") pod \"lws-controller-manager-667dbb5fb5-ngctn\" (UID: \"3ed95d68-12dc-4162-a244-6113aeff38ba\") " pod="openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn"
Sep 15 11:31:04.602205 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.602169    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn"
Sep 15 11:31:04.876841 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:04.876768    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn"]
Sep 15 11:31:04.879726 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:31:04.879694    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed95d68_12dc_4162_a244_6113aeff38ba.slice/crio-6da2771259cb758c4ec47b17402f2d31cf0df81e447175c39437d643628b8b15 WatchSource:0}: Error finding container 6da2771259cb758c4ec47b17402f2d31cf0df81e447175c39437d643628b8b15: Status 404 returned error can't find the container with id 6da2771259cb758c4ec47b17402f2d31cf0df81e447175c39437d643628b8b15
Sep 15 11:31:05.370012 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:05.369970    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn" event={"ID":"3ed95d68-12dc-4162-a244-6113aeff38ba","Type":"ContainerStarted","Data":"6da2771259cb758c4ec47b17402f2d31cf0df81e447175c39437d643628b8b15"}
Sep 15 11:31:08.383985 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:08.383948    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn" event={"ID":"3ed95d68-12dc-4162-a244-6113aeff38ba","Type":"ContainerStarted","Data":"76f68b69aa6ade06bc94b62c4610f199fb12a63a2beb46f3ee90d91606ca2da7"}
Sep 15 11:31:08.384391 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:08.384187    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn"
Sep 15 11:31:08.409511 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:08.409461    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn" podStartSLOduration=1.8328576760000002 podStartE2EDuration="4.409445479s" totalImagesPullingTime="2.576587803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:31:08.409139604 +0000 UTC m=+367.071634804" watchObservedRunningTime="2026-09-15 11:31:08.409445479 +0000 UTC m=+367.071940660"
Sep 15 11:31:10.006638 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:10.006603    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-strq5"]
Sep 15 11:31:10.009876 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:10.009855    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-strq5"
Sep 15 11:31:10.015424 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:10.015403    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-l5tvk\""
Sep 15 11:31:10.015635 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:10.015621    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 15 11:31:10.032576 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:10.032548    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-strq5"]
Sep 15 11:31:10.130447 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:10.130410    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc794b58-48a4-4697-a523-7e46fcf3464a-cert\") pod \"kserve-controller-manager-d6f4b65b8-strq5\" (UID: \"fc794b58-48a4-4697-a523-7e46fcf3464a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-strq5"
Sep 15 11:31:10.130447 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:10.130459    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkn8h\" (UniqueName: \"kubernetes.io/projected/fc794b58-48a4-4697-a523-7e46fcf3464a-kube-api-access-bkn8h\") pod \"kserve-controller-manager-d6f4b65b8-strq5\" (UID: \"fc794b58-48a4-4697-a523-7e46fcf3464a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-strq5"
Sep 15 11:31:10.231969 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:10.231573    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkn8h\" (UniqueName: \"kubernetes.io/projected/fc794b58-48a4-4697-a523-7e46fcf3464a-kube-api-access-bkn8h\") pod \"kserve-controller-manager-d6f4b65b8-strq5\" (UID: \"fc794b58-48a4-4697-a523-7e46fcf3464a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-strq5"
Sep 15 11:31:10.236076 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:10.232315    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc794b58-48a4-4697-a523-7e46fcf3464a-cert\") pod \"kserve-controller-manager-d6f4b65b8-strq5\" (UID: \"fc794b58-48a4-4697-a523-7e46fcf3464a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-strq5"
Sep 15 11:31:10.236076 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:31:10.232499    2562 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 15 11:31:10.236076 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:31:10.232571    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc794b58-48a4-4697-a523-7e46fcf3464a-cert podName:fc794b58-48a4-4697-a523-7e46fcf3464a nodeName:}" failed. No retries permitted until 2026-09-15 11:31:10.732550383 +0000 UTC m=+369.395045559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fc794b58-48a4-4697-a523-7e46fcf3464a-cert") pod "kserve-controller-manager-d6f4b65b8-strq5" (UID: "fc794b58-48a4-4697-a523-7e46fcf3464a") : secret "kserve-webhook-server-cert" not found
Sep 15 11:31:10.243110 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:10.243074    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkn8h\" (UniqueName: \"kubernetes.io/projected/fc794b58-48a4-4697-a523-7e46fcf3464a-kube-api-access-bkn8h\") pod \"kserve-controller-manager-d6f4b65b8-strq5\" (UID: \"fc794b58-48a4-4697-a523-7e46fcf3464a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-strq5"
Sep 15 11:31:10.735938 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:10.735897    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc794b58-48a4-4697-a523-7e46fcf3464a-cert\") pod \"kserve-controller-manager-d6f4b65b8-strq5\" (UID: \"fc794b58-48a4-4697-a523-7e46fcf3464a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-strq5"
Sep 15 11:31:10.738382 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:10.738360    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc794b58-48a4-4697-a523-7e46fcf3464a-cert\") pod \"kserve-controller-manager-d6f4b65b8-strq5\" (UID: \"fc794b58-48a4-4697-a523-7e46fcf3464a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-strq5"
Sep 15 11:31:10.919531 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:10.919490    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-strq5"
Sep 15 11:31:11.198821 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:11.198797    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-strq5"]
Sep 15 11:31:11.200790 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:31:11.200759    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc794b58_48a4_4697_a523_7e46fcf3464a.slice/crio-baf110a04d16f4c7b602608d3a882ab5d1ca666ca8a983c03dbedf9a7dbb912a WatchSource:0}: Error finding container baf110a04d16f4c7b602608d3a882ab5d1ca666ca8a983c03dbedf9a7dbb912a: Status 404 returned error can't find the container with id baf110a04d16f4c7b602608d3a882ab5d1ca666ca8a983c03dbedf9a7dbb912a
Sep 15 11:31:11.395886 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:11.395849    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-strq5" event={"ID":"fc794b58-48a4-4697-a523-7e46fcf3464a","Type":"ContainerStarted","Data":"baf110a04d16f4c7b602608d3a882ab5d1ca666ca8a983c03dbedf9a7dbb912a"}
Sep 15 11:31:14.410950 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:14.410908    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-strq5" event={"ID":"fc794b58-48a4-4697-a523-7e46fcf3464a","Type":"ContainerStarted","Data":"5f4b4409e233d3d9642e2fb39f0ae539495c1c4aedc3c3d4f0bf68cf83d1ce60"}
Sep 15 11:31:14.411443 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:14.411030    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-strq5"
Sep 15 11:31:14.438062 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:14.437978    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-strq5" podStartSLOduration=2.856542246 podStartE2EDuration="5.437955972s" totalImagesPullingTime="2.581413726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:31:14.435201413 +0000 UTC m=+373.097696607" watchObservedRunningTime="2026-09-15 11:31:14.437955972 +0000 UTC m=+373.100451154"
Sep 15 11:31:19.390739 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:19.390703    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-667dbb5fb5-ngctn"
Sep 15 11:31:45.420124 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:31:45.420079    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-strq5"
Sep 15 11:32:24.698561 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:24.698479    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn"]
Sep 15 11:32:24.701640 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:24.701622    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn"
Sep 15 11:32:24.705234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:24.705205    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lmwgr\""
Sep 15 11:32:24.705379 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:24.705234    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 11:32:24.705720 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:24.705706    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 11:32:24.714866 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:24.714843    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn"]
Sep 15 11:32:24.820732 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:24.820695    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-952bd\" (UniqueName: \"kubernetes.io/projected/6a1b67a0-425a-4478-a9af-dfdfeb0a5cce-kube-api-access-952bd\") pod \"limitador-operator-controller-manager-74495c69f8-ccwxn\" (UID: \"6a1b67a0-425a-4478-a9af-dfdfeb0a5cce\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn"
Sep 15 11:32:24.921873 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:24.921830    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-952bd\" (UniqueName: \"kubernetes.io/projected/6a1b67a0-425a-4478-a9af-dfdfeb0a5cce-kube-api-access-952bd\") pod \"limitador-operator-controller-manager-74495c69f8-ccwxn\" (UID: \"6a1b67a0-425a-4478-a9af-dfdfeb0a5cce\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn"
Sep 15 11:32:24.930931 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:24.930897    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-952bd\" (UniqueName: \"kubernetes.io/projected/6a1b67a0-425a-4478-a9af-dfdfeb0a5cce-kube-api-access-952bd\") pod \"limitador-operator-controller-manager-74495c69f8-ccwxn\" (UID: \"6a1b67a0-425a-4478-a9af-dfdfeb0a5cce\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn"
Sep 15 11:32:25.011397 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:25.011300    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn"
Sep 15 11:32:25.286872 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:25.286847    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn"]
Sep 15 11:32:25.289545 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:32:25.289516    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1b67a0_425a_4478_a9af_dfdfeb0a5cce.slice/crio-637b766c7a749b7ccf827f61b947e4f16e6697052c8c6ef6247856fa5a7076f4 WatchSource:0}: Error finding container 637b766c7a749b7ccf827f61b947e4f16e6697052c8c6ef6247856fa5a7076f4: Status 404 returned error can't find the container with id 637b766c7a749b7ccf827f61b947e4f16e6697052c8c6ef6247856fa5a7076f4
Sep 15 11:32:25.691040 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:25.690999    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn" event={"ID":"6a1b67a0-425a-4478-a9af-dfdfeb0a5cce","Type":"ContainerStarted","Data":"637b766c7a749b7ccf827f61b947e4f16e6697052c8c6ef6247856fa5a7076f4"}
Sep 15 11:32:27.133422 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:27.133387    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xdz2w"]
Sep 15 11:32:27.142150 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:27.142122    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xdz2w"
Sep 15 11:32:27.144604 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:27.144535    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 15 11:32:27.144604 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:27.144601    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 15 11:32:27.145338 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:27.145312    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bgv4b\""
Sep 15 11:32:27.146970 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:27.146946    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xdz2w"]
Sep 15 11:32:27.242420 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:27.242384    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/beb108cb-5872-4cb9-8733-9dceae10f816-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xdz2w\" (UID: \"beb108cb-5872-4cb9-8733-9dceae10f816\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xdz2w"
Sep 15 11:32:27.242589 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:27.242487    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/beb108cb-5872-4cb9-8733-9dceae10f816-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xdz2w\" (UID: \"beb108cb-5872-4cb9-8733-9dceae10f816\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xdz2w"
Sep 15 11:32:27.242589 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:27.242529    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sggw\" (UniqueName: \"kubernetes.io/projected/beb108cb-5872-4cb9-8733-9dceae10f816-kube-api-access-8sggw\") pod \"kuadrant-console-plugin-fbdd4f445-xdz2w\" (UID: \"beb108cb-5872-4cb9-8733-9dceae10f816\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xdz2w"
Sep 15 11:32:27.344071 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:27.344019    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/beb108cb-5872-4cb9-8733-9dceae10f816-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xdz2w\" (UID: \"beb108cb-5872-4cb9-8733-9dceae10f816\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xdz2w"
Sep 15 11:32:27.344263 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:27.344095    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sggw\" (UniqueName: \"kubernetes.io/projected/beb108cb-5872-4cb9-8733-9dceae10f816-kube-api-access-8sggw\") pod \"kuadrant-console-plugin-fbdd4f445-xdz2w\" (UID: \"beb108cb-5872-4cb9-8733-9dceae10f816\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xdz2w"
Sep 15 11:32:27.344263 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:27.344154    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/beb108cb-5872-4cb9-8733-9dceae10f816-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xdz2w\" (UID: \"beb108cb-5872-4cb9-8733-9dceae10f816\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xdz2w"
Sep 15 11:32:27.344263 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:32:27.344216    2562 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 15 11:32:27.344434 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:32:27.344307    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beb108cb-5872-4cb9-8733-9dceae10f816-plugin-serving-cert podName:beb108cb-5872-4cb9-8733-9dceae10f816 nodeName:}" failed. No retries permitted until 2026-09-15 11:32:27.844276819 +0000 UTC m=+446.506771981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/beb108cb-5872-4cb9-8733-9dceae10f816-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-xdz2w" (UID: "beb108cb-5872-4cb9-8733-9dceae10f816") : secret "plugin-serving-cert" not found
Sep 15 11:32:27.344953 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:27.344928    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/beb108cb-5872-4cb9-8733-9dceae10f816-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xdz2w\" (UID: \"beb108cb-5872-4cb9-8733-9dceae10f816\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xdz2w"
Sep 15 11:32:27.355753 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:27.355730    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sggw\" (UniqueName: \"kubernetes.io/projected/beb108cb-5872-4cb9-8733-9dceae10f816-kube-api-access-8sggw\") pod \"kuadrant-console-plugin-fbdd4f445-xdz2w\" (UID: \"beb108cb-5872-4cb9-8733-9dceae10f816\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xdz2w"
Sep 15 11:32:27.850037 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:27.850002    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/beb108cb-5872-4cb9-8733-9dceae10f816-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xdz2w\" (UID: \"beb108cb-5872-4cb9-8733-9dceae10f816\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xdz2w"
Sep 15 11:32:27.852770 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:27.852741    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/beb108cb-5872-4cb9-8733-9dceae10f816-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xdz2w\" (UID: \"beb108cb-5872-4cb9-8733-9dceae10f816\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xdz2w"
Sep 15 11:32:28.054378 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:28.054290    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xdz2w"
Sep 15 11:32:28.327045 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:28.326971    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xdz2w"]
Sep 15 11:32:28.329835 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:32:28.329810    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbeb108cb_5872_4cb9_8733_9dceae10f816.slice/crio-deebf70aef690ffbba3c4f275e0840af47a70aeef0bd0fd13831fbbe571784a8 WatchSource:0}: Error finding container deebf70aef690ffbba3c4f275e0840af47a70aeef0bd0fd13831fbbe571784a8: Status 404 returned error can't find the container with id deebf70aef690ffbba3c4f275e0840af47a70aeef0bd0fd13831fbbe571784a8
Sep 15 11:32:28.709303 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:28.709262    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn" event={"ID":"6a1b67a0-425a-4478-a9af-dfdfeb0a5cce","Type":"ContainerStarted","Data":"07ef7677343e807cd06a05318d122b2862fa7e1fc843b145245e9415cf1d8a07"}
Sep 15 11:32:28.709504 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:28.709382    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn"
Sep 15 11:32:28.710563 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:28.710541    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xdz2w" event={"ID":"beb108cb-5872-4cb9-8733-9dceae10f816","Type":"ContainerStarted","Data":"deebf70aef690ffbba3c4f275e0840af47a70aeef0bd0fd13831fbbe571784a8"}
Sep 15 11:32:28.727564 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:28.727515    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn" podStartSLOduration=2.216638914 podStartE2EDuration="4.727496504s" totalImagesPullingTime="2.51085759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:32:28.726850022 +0000 UTC m=+447.389345237" watchObservedRunningTime="2026-09-15 11:32:28.727496504 +0000 UTC m=+447.389991686"
Sep 15 11:32:33.739520 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:33.739426    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xdz2w" event={"ID":"beb108cb-5872-4cb9-8733-9dceae10f816","Type":"ContainerStarted","Data":"e32ff767d2ddefc5a9f243405ba12800d028609adcccd1c78003fc051c9b2fd7"}
Sep 15 11:32:33.757569 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:33.757513    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xdz2w" podStartSLOduration=1.739277089 podStartE2EDuration="6.757496072s" totalImagesPullingTime="5.018218983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:32:33.755584868 +0000 UTC m=+452.418080049" watchObservedRunningTime="2026-09-15 11:32:33.757496072 +0000 UTC m=+452.419991253"
Sep 15 11:32:38.129712 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.129678    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn"]
Sep 15 11:32:38.130136 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.129902    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn" podUID="6a1b67a0-425a-4478-a9af-dfdfeb0a5cce" containerName="manager" containerID="cri-o://07ef7677343e807cd06a05318d122b2862fa7e1fc843b145245e9415cf1d8a07" gracePeriod=2
Sep 15 11:32:38.131958 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.131928    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn"
Sep 15 11:32:38.135946 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:32:38.135898    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ccwxn\" is forbidden: User \"system:node:ip-10-0-135-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-143.ec2.internal' and this object" podUID="6a1b67a0-425a-4478-a9af-dfdfeb0a5cce" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn"
Sep 15 11:32:38.136288 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.136259    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn"]
Sep 15 11:32:38.165196 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.165168    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qdr85"]
Sep 15 11:32:38.165475 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.165461    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a1b67a0-425a-4478-a9af-dfdfeb0a5cce" containerName="manager"
Sep 15 11:32:38.165537 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.165477    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a1b67a0-425a-4478-a9af-dfdfeb0a5cce" containerName="manager"
Sep 15 11:32:38.165537 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.165535    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a1b67a0-425a-4478-a9af-dfdfeb0a5cce" containerName="manager"
Sep 15 11:32:38.168339 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.168322    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qdr85"
Sep 15 11:32:38.170145 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:32:38.170110    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ccwxn\" is forbidden: User \"system:node:ip-10-0-135-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-143.ec2.internal' and this object" podUID="6a1b67a0-425a-4478-a9af-dfdfeb0a5cce" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn"
Sep 15 11:32:38.190039 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.190007    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qdr85"]
Sep 15 11:32:38.227344 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.227311    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7d7h\" (UniqueName: \"kubernetes.io/projected/e43941e0-3e20-4744-98b1-55855559ca7e-kube-api-access-p7d7h\") pod \"limitador-operator-controller-manager-74495c69f8-qdr85\" (UID: \"e43941e0-3e20-4744-98b1-55855559ca7e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qdr85"
Sep 15 11:32:38.328255 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.328222    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7d7h\" (UniqueName: \"kubernetes.io/projected/e43941e0-3e20-4744-98b1-55855559ca7e-kube-api-access-p7d7h\") pod \"limitador-operator-controller-manager-74495c69f8-qdr85\" (UID: \"e43941e0-3e20-4744-98b1-55855559ca7e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qdr85"
Sep 15 11:32:38.343403 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.343362    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7d7h\" (UniqueName: \"kubernetes.io/projected/e43941e0-3e20-4744-98b1-55855559ca7e-kube-api-access-p7d7h\") pod \"limitador-operator-controller-manager-74495c69f8-qdr85\" (UID: \"e43941e0-3e20-4744-98b1-55855559ca7e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qdr85"
Sep 15 11:32:38.380386 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.380322    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn"
Sep 15 11:32:38.382427 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:32:38.382396    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ccwxn\" is forbidden: User \"system:node:ip-10-0-135-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-143.ec2.internal' and this object" podUID="6a1b67a0-425a-4478-a9af-dfdfeb0a5cce" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn"
Sep 15 11:32:38.428943 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.428913    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-952bd\" (UniqueName: \"kubernetes.io/projected/6a1b67a0-425a-4478-a9af-dfdfeb0a5cce-kube-api-access-952bd\") pod \"6a1b67a0-425a-4478-a9af-dfdfeb0a5cce\" (UID: \"6a1b67a0-425a-4478-a9af-dfdfeb0a5cce\") "
Sep 15 11:32:38.431201 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.431166    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1b67a0-425a-4478-a9af-dfdfeb0a5cce-kube-api-access-952bd" (OuterVolumeSpecName: "kube-api-access-952bd") pod "6a1b67a0-425a-4478-a9af-dfdfeb0a5cce" (UID: "6a1b67a0-425a-4478-a9af-dfdfeb0a5cce"). InnerVolumeSpecName "kube-api-access-952bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:32:38.519135 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.519094    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qdr85"
Sep 15 11:32:38.529352 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.529322    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-952bd\" (UniqueName: \"kubernetes.io/projected/6a1b67a0-425a-4478-a9af-dfdfeb0a5cce-kube-api-access-952bd\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:32:38.763626 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.761470    2562 generic.go:356] "Generic (PLEG): container finished" podID="6a1b67a0-425a-4478-a9af-dfdfeb0a5cce" containerID="07ef7677343e807cd06a05318d122b2862fa7e1fc843b145245e9415cf1d8a07" exitCode=0
Sep 15 11:32:38.763626 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.761588    2562 scope.go:117] "RemoveContainer" containerID="07ef7677343e807cd06a05318d122b2862fa7e1fc843b145245e9415cf1d8a07"
Sep 15 11:32:38.763626 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.761744    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn"
Sep 15 11:32:38.765632 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:32:38.765588    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ccwxn\" is forbidden: User \"system:node:ip-10-0-135-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-143.ec2.internal' and this object" podUID="6a1b67a0-425a-4478-a9af-dfdfeb0a5cce" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn"
Sep 15 11:32:38.772943 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.772922    2562 scope.go:117] "RemoveContainer" containerID="07ef7677343e807cd06a05318d122b2862fa7e1fc843b145245e9415cf1d8a07"
Sep 15 11:32:38.773317 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:32:38.773295    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ef7677343e807cd06a05318d122b2862fa7e1fc843b145245e9415cf1d8a07\": container with ID starting with 07ef7677343e807cd06a05318d122b2862fa7e1fc843b145245e9415cf1d8a07 not found: ID does not exist" containerID="07ef7677343e807cd06a05318d122b2862fa7e1fc843b145245e9415cf1d8a07"
Sep 15 11:32:38.773451 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.773325    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ef7677343e807cd06a05318d122b2862fa7e1fc843b145245e9415cf1d8a07"} err="failed to get container status \"07ef7677343e807cd06a05318d122b2862fa7e1fc843b145245e9415cf1d8a07\": rpc error: code = NotFound desc = could not find container \"07ef7677343e807cd06a05318d122b2862fa7e1fc843b145245e9415cf1d8a07\": container with ID starting with 07ef7677343e807cd06a05318d122b2862fa7e1fc843b145245e9415cf1d8a07 not found: ID does not exist"
Sep 15 11:32:38.775190 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:32:38.775165    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ccwxn\" is forbidden: User \"system:node:ip-10-0-135-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-143.ec2.internal' and this object" podUID="6a1b67a0-425a-4478-a9af-dfdfeb0a5cce" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn"
Sep 15 11:32:38.791849 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:38.791824    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qdr85"]
Sep 15 11:32:38.795310 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:32:38.795267    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43941e0_3e20_4744_98b1_55855559ca7e.slice/crio-13c85a5b9e642c1dc942d62ef17c87254c87671d0c075cb4116e452eb80813de WatchSource:0}: Error finding container 13c85a5b9e642c1dc942d62ef17c87254c87671d0c075cb4116e452eb80813de: Status 404 returned error can't find the container with id 13c85a5b9e642c1dc942d62ef17c87254c87671d0c075cb4116e452eb80813de
Sep 15 11:32:39.768047 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:39.768010    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qdr85" event={"ID":"e43941e0-3e20-4744-98b1-55855559ca7e","Type":"ContainerStarted","Data":"b90e8b65c5f9b56a573ab312f327d4a3b2187c77ea211950c41eca070197b44f"}
Sep 15 11:32:39.768550 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:39.768082    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qdr85" event={"ID":"e43941e0-3e20-4744-98b1-55855559ca7e","Type":"ContainerStarted","Data":"13c85a5b9e642c1dc942d62ef17c87254c87671d0c075cb4116e452eb80813de"}
Sep 15 11:32:39.768550 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:39.768138    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qdr85"
Sep 15 11:32:39.770128 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:32:39.770101    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ccwxn\" is forbidden: User \"system:node:ip-10-0-135-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-143.ec2.internal' and this object" podUID="6a1b67a0-425a-4478-a9af-dfdfeb0a5cce" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ccwxn"
Sep 15 11:32:39.786120 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:39.786073    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qdr85" podStartSLOduration=1.7860436320000002 podStartE2EDuration="1.786043632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:32:39.784633111 +0000 UTC m=+458.447128291" watchObservedRunningTime="2026-09-15 11:32:39.786043632 +0000 UTC m=+458.448538813"
Sep 15 11:32:39.873535 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:39.873499    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a1b67a0-425a-4478-a9af-dfdfeb0a5cce" path="/var/lib/kubelet/pods/6a1b67a0-425a-4478-a9af-dfdfeb0a5cce/volumes"
Sep 15 11:32:50.774197 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:32:50.774166    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qdr85"
Sep 15 11:33:11.614809 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:11.614768    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:33:11.632428 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:11.632392    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:33:11.632575 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:11.632536    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-mgpww"
Sep 15 11:33:11.635216 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:11.635189    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 15 11:33:11.651081 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:11.651038    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:33:11.655150 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:11.655126    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/907a5c3b-ceb5-43d3-96c4-c4ba68f8f613-config-file\") pod \"limitador-limitador-84994f7ffc-mgpww\" (UID: \"907a5c3b-ceb5-43d3-96c4-c4ba68f8f613\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mgpww"
Sep 15 11:33:11.655266 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:11.655215    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9qwb\" (UniqueName: \"kubernetes.io/projected/907a5c3b-ceb5-43d3-96c4-c4ba68f8f613-kube-api-access-n9qwb\") pod \"limitador-limitador-84994f7ffc-mgpww\" (UID: \"907a5c3b-ceb5-43d3-96c4-c4ba68f8f613\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mgpww"
Sep 15 11:33:11.755687 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:11.755642    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/907a5c3b-ceb5-43d3-96c4-c4ba68f8f613-config-file\") pod \"limitador-limitador-84994f7ffc-mgpww\" (UID: \"907a5c3b-ceb5-43d3-96c4-c4ba68f8f613\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mgpww"
Sep 15 11:33:11.755851 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:11.755718    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9qwb\" (UniqueName: \"kubernetes.io/projected/907a5c3b-ceb5-43d3-96c4-c4ba68f8f613-kube-api-access-n9qwb\") pod \"limitador-limitador-84994f7ffc-mgpww\" (UID: \"907a5c3b-ceb5-43d3-96c4-c4ba68f8f613\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mgpww"
Sep 15 11:33:11.756341 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:11.756318    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/907a5c3b-ceb5-43d3-96c4-c4ba68f8f613-config-file\") pod \"limitador-limitador-84994f7ffc-mgpww\" (UID: \"907a5c3b-ceb5-43d3-96c4-c4ba68f8f613\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mgpww"
Sep 15 11:33:11.763821 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:11.763796    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9qwb\" (UniqueName: \"kubernetes.io/projected/907a5c3b-ceb5-43d3-96c4-c4ba68f8f613-kube-api-access-n9qwb\") pod \"limitador-limitador-84994f7ffc-mgpww\" (UID: \"907a5c3b-ceb5-43d3-96c4-c4ba68f8f613\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mgpww"
Sep 15 11:33:11.942231 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:11.942137    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-mgpww"
Sep 15 11:33:12.171202 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:12.171166    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-qct94"]
Sep 15 11:33:12.174092 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:12.174050    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qct94"
Sep 15 11:33:12.176288 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:12.176267    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wlpnn\""
Sep 15 11:33:12.181598 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:12.181575    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qct94"]
Sep 15 11:33:12.211619 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:12.211593    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:33:12.214527 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:33:12.214433    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907a5c3b_ceb5_43d3_96c4_c4ba68f8f613.slice/crio-c19624d346f26a3b343a22652e2beeb9065ef22c8e9501f63031efcc15bb411e WatchSource:0}: Error finding container c19624d346f26a3b343a22652e2beeb9065ef22c8e9501f63031efcc15bb411e: Status 404 returned error can't find the container with id c19624d346f26a3b343a22652e2beeb9065ef22c8e9501f63031efcc15bb411e
Sep 15 11:33:12.259516 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:12.259474    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6489m\" (UniqueName: \"kubernetes.io/projected/482c94c6-22da-4772-b35e-1e0bd7106afd-kube-api-access-6489m\") pod \"authorino-59c974bd48-qct94\" (UID: \"482c94c6-22da-4772-b35e-1e0bd7106afd\") " pod="kuadrant-system/authorino-59c974bd48-qct94"
Sep 15 11:33:12.360753 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:12.360712    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6489m\" (UniqueName: \"kubernetes.io/projected/482c94c6-22da-4772-b35e-1e0bd7106afd-kube-api-access-6489m\") pod \"authorino-59c974bd48-qct94\" (UID: \"482c94c6-22da-4772-b35e-1e0bd7106afd\") " pod="kuadrant-system/authorino-59c974bd48-qct94"
Sep 15 11:33:12.369251 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:12.369219    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6489m\" (UniqueName: \"kubernetes.io/projected/482c94c6-22da-4772-b35e-1e0bd7106afd-kube-api-access-6489m\") pod \"authorino-59c974bd48-qct94\" (UID: \"482c94c6-22da-4772-b35e-1e0bd7106afd\") " pod="kuadrant-system/authorino-59c974bd48-qct94"
Sep 15 11:33:12.483956 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:12.483860    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qct94"
Sep 15 11:33:12.754666 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:12.754576    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qct94"]
Sep 15 11:33:12.759275 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:33:12.759246    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod482c94c6_22da_4772_b35e_1e0bd7106afd.slice/crio-c9fb0ad98051553e6c39dbd82a9a7e20cb8c12b23bc7308f056a6cbd03f54d74 WatchSource:0}: Error finding container c9fb0ad98051553e6c39dbd82a9a7e20cb8c12b23bc7308f056a6cbd03f54d74: Status 404 returned error can't find the container with id c9fb0ad98051553e6c39dbd82a9a7e20cb8c12b23bc7308f056a6cbd03f54d74
Sep 15 11:33:12.900074 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:12.900022    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-mgpww" event={"ID":"907a5c3b-ceb5-43d3-96c4-c4ba68f8f613","Type":"ContainerStarted","Data":"c19624d346f26a3b343a22652e2beeb9065ef22c8e9501f63031efcc15bb411e"}
Sep 15 11:33:12.901465 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:12.901433    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qct94" event={"ID":"482c94c6-22da-4772-b35e-1e0bd7106afd","Type":"ContainerStarted","Data":"c9fb0ad98051553e6c39dbd82a9a7e20cb8c12b23bc7308f056a6cbd03f54d74"}
Sep 15 11:33:13.907692 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:13.907652    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-mgpww" event={"ID":"907a5c3b-ceb5-43d3-96c4-c4ba68f8f613","Type":"ContainerStarted","Data":"5af1d2931294d1b606d1c7da2837a498e4f7605df9ff3b369eac653649eaf67e"}
Sep 15 11:33:13.908173 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:13.907722    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-mgpww"
Sep 15 11:33:13.927143 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:13.927086    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-mgpww" podStartSLOduration=1.992911866 podStartE2EDuration="2.927049187s" totalImagesPullingTime="934.137321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:33:13.925369941 +0000 UTC m=+492.587865145" watchObservedRunningTime="2026-09-15 11:33:13.927049187 +0000 UTC m=+492.589544370"
Sep 15 11:33:14.913637 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:14.913599    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qct94" event={"ID":"482c94c6-22da-4772-b35e-1e0bd7106afd","Type":"ContainerStarted","Data":"ee5284bef73434ea687baedd184f5ff21875d9406fae98e8de21aafbbc81886a"}
Sep 15 11:33:14.930572 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:14.930526    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-qct94" podStartSLOduration=1.182017076 podStartE2EDuration="2.930508821s" totalImagesPullingTime="1.748491745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:33:14.929682945 +0000 UTC m=+493.592178128" watchObservedRunningTime="2026-09-15 11:33:14.930508821 +0000 UTC m=+493.593004002"
Sep 15 11:33:24.915030 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:24.914999    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-mgpww"
Sep 15 11:33:41.843560 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:41.843523    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qct94"]
Sep 15 11:33:41.844010 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:41.843741    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-qct94" podUID="482c94c6-22da-4772-b35e-1e0bd7106afd" containerName="authorino" containerID="cri-o://ee5284bef73434ea687baedd184f5ff21875d9406fae98e8de21aafbbc81886a" gracePeriod=30
Sep 15 11:33:42.021868 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:42.021833    2562 generic.go:356] "Generic (PLEG): container finished" podID="482c94c6-22da-4772-b35e-1e0bd7106afd" containerID="ee5284bef73434ea687baedd184f5ff21875d9406fae98e8de21aafbbc81886a" exitCode=0
Sep 15 11:33:42.021997 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:42.021902    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qct94" event={"ID":"482c94c6-22da-4772-b35e-1e0bd7106afd","Type":"ContainerDied","Data":"ee5284bef73434ea687baedd184f5ff21875d9406fae98e8de21aafbbc81886a"}
Sep 15 11:33:42.094426 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:42.094369    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qct94"
Sep 15 11:33:42.179242 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:42.179200    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6489m\" (UniqueName: \"kubernetes.io/projected/482c94c6-22da-4772-b35e-1e0bd7106afd-kube-api-access-6489m\") pod \"482c94c6-22da-4772-b35e-1e0bd7106afd\" (UID: \"482c94c6-22da-4772-b35e-1e0bd7106afd\") "
Sep 15 11:33:42.181284 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:42.181258    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482c94c6-22da-4772-b35e-1e0bd7106afd-kube-api-access-6489m" (OuterVolumeSpecName: "kube-api-access-6489m") pod "482c94c6-22da-4772-b35e-1e0bd7106afd" (UID: "482c94c6-22da-4772-b35e-1e0bd7106afd"). InnerVolumeSpecName "kube-api-access-6489m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:33:42.280724 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:42.280691    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6489m\" (UniqueName: \"kubernetes.io/projected/482c94c6-22da-4772-b35e-1e0bd7106afd-kube-api-access-6489m\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:33:43.027825 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:43.027798    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qct94"
Sep 15 11:33:43.028286 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:43.027802    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qct94" event={"ID":"482c94c6-22da-4772-b35e-1e0bd7106afd","Type":"ContainerDied","Data":"c9fb0ad98051553e6c39dbd82a9a7e20cb8c12b23bc7308f056a6cbd03f54d74"}
Sep 15 11:33:43.028286 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:43.027913    2562 scope.go:117] "RemoveContainer" containerID="ee5284bef73434ea687baedd184f5ff21875d9406fae98e8de21aafbbc81886a"
Sep 15 11:33:43.049838 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:43.049811    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qct94"]
Sep 15 11:33:43.054973 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:43.054951    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qct94"]
Sep 15 11:33:43.873837 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:33:43.873807    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="482c94c6-22da-4772-b35e-1e0bd7106afd" path="/var/lib/kubelet/pods/482c94c6-22da-4772-b35e-1e0bd7106afd/volumes"
Sep 15 11:34:22.131667 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:22.131628    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:34:46.952986 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:46.952951    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"]
Sep 15 11:34:46.953467 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:46.953263    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="482c94c6-22da-4772-b35e-1e0bd7106afd" containerName="authorino"
Sep 15 11:34:46.953467 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:46.953275    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="482c94c6-22da-4772-b35e-1e0bd7106afd" containerName="authorino"
Sep 15 11:34:46.953467 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:46.953326    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="482c94c6-22da-4772-b35e-1e0bd7106afd" containerName="authorino"
Sep 15 11:34:46.956192 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:46.956174    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:46.958396 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:46.958370    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 15 11:34:46.958396 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:46.958376    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 11:34:46.958992 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:46.958976    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ghj57\""
Sep 15 11:34:46.959047 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:46.959015    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 11:34:46.966303 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:46.966281    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"]
Sep 15 11:34:46.996222 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:46.996189    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8256e21e-62b3-4003-b940-21ba1ca1e0bc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr\" (UID: \"8256e21e-62b3-4003-b940-21ba1ca1e0bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:46.996370 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:46.996242    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8256e21e-62b3-4003-b940-21ba1ca1e0bc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr\" (UID: \"8256e21e-62b3-4003-b940-21ba1ca1e0bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:46.996370 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:46.996367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8256e21e-62b3-4003-b940-21ba1ca1e0bc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr\" (UID: \"8256e21e-62b3-4003-b940-21ba1ca1e0bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:46.996464 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:46.996405    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8256e21e-62b3-4003-b940-21ba1ca1e0bc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr\" (UID: \"8256e21e-62b3-4003-b940-21ba1ca1e0bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:46.996464 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:46.996426    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8256e21e-62b3-4003-b940-21ba1ca1e0bc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr\" (UID: \"8256e21e-62b3-4003-b940-21ba1ca1e0bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:46.996582 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:46.996559    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcj8d\" (UniqueName: \"kubernetes.io/projected/8256e21e-62b3-4003-b940-21ba1ca1e0bc-kube-api-access-mcj8d\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr\" (UID: \"8256e21e-62b3-4003-b940-21ba1ca1e0bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:47.097827 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:47.097788    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8256e21e-62b3-4003-b940-21ba1ca1e0bc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr\" (UID: \"8256e21e-62b3-4003-b940-21ba1ca1e0bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:47.097827 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:47.097828    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8256e21e-62b3-4003-b940-21ba1ca1e0bc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr\" (UID: \"8256e21e-62b3-4003-b940-21ba1ca1e0bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:47.098048 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:47.097859    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8256e21e-62b3-4003-b940-21ba1ca1e0bc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr\" (UID: \"8256e21e-62b3-4003-b940-21ba1ca1e0bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:47.098048 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:47.097886    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8256e21e-62b3-4003-b940-21ba1ca1e0bc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr\" (UID: \"8256e21e-62b3-4003-b940-21ba1ca1e0bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:47.098048 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:47.097903    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8256e21e-62b3-4003-b940-21ba1ca1e0bc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr\" (UID: \"8256e21e-62b3-4003-b940-21ba1ca1e0bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:47.098048 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:47.097926    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcj8d\" (UniqueName: \"kubernetes.io/projected/8256e21e-62b3-4003-b940-21ba1ca1e0bc-kube-api-access-mcj8d\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr\" (UID: \"8256e21e-62b3-4003-b940-21ba1ca1e0bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:47.098336 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:47.098313    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8256e21e-62b3-4003-b940-21ba1ca1e0bc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr\" (UID: \"8256e21e-62b3-4003-b940-21ba1ca1e0bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:47.098409 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:47.098371    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8256e21e-62b3-4003-b940-21ba1ca1e0bc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr\" (UID: \"8256e21e-62b3-4003-b940-21ba1ca1e0bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:47.098461 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:47.098412    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8256e21e-62b3-4003-b940-21ba1ca1e0bc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr\" (UID: \"8256e21e-62b3-4003-b940-21ba1ca1e0bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:47.100044 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:47.100022    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8256e21e-62b3-4003-b940-21ba1ca1e0bc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr\" (UID: \"8256e21e-62b3-4003-b940-21ba1ca1e0bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:47.100397 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:47.100380    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8256e21e-62b3-4003-b940-21ba1ca1e0bc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr\" (UID: \"8256e21e-62b3-4003-b940-21ba1ca1e0bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:47.105803 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:47.105784    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcj8d\" (UniqueName: \"kubernetes.io/projected/8256e21e-62b3-4003-b940-21ba1ca1e0bc-kube-api-access-mcj8d\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr\" (UID: \"8256e21e-62b3-4003-b940-21ba1ca1e0bc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:47.266626 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:47.266531    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:47.539874 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:47.539786    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"]
Sep 15 11:34:47.543126 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:34:47.543097    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8256e21e_62b3_4003_b940_21ba1ca1e0bc.slice/crio-65a3a6c048e437ebaeddcac231997c676147a3dd30d42fb8edfba26da096866a WatchSource:0}: Error finding container 65a3a6c048e437ebaeddcac231997c676147a3dd30d42fb8edfba26da096866a: Status 404 returned error can't find the container with id 65a3a6c048e437ebaeddcac231997c676147a3dd30d42fb8edfba26da096866a
Sep 15 11:34:47.566598 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:47.566571    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:34:48.300574 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:48.300527    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr" event={"ID":"8256e21e-62b3-4003-b940-21ba1ca1e0bc","Type":"ContainerStarted","Data":"65a3a6c048e437ebaeddcac231997c676147a3dd30d42fb8edfba26da096866a"}
Sep 15 11:34:49.305760 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:49.305724    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr" event={"ID":"8256e21e-62b3-4003-b940-21ba1ca1e0bc","Type":"ContainerStarted","Data":"c201f801d9cf162dc801939a1488ded9bed01d7a30d0d0ffa515b40b3c00a055"}
Sep 15 11:34:49.323073 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:49.322994    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr" podStartSLOduration=2.331421095 podStartE2EDuration="3.322974783s" totalImagesPullingTime="991.553688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:34:49.322639759 +0000 UTC m=+587.985134940" watchObservedRunningTime="2026-09-15 11:34:49.322974783 +0000 UTC m=+587.985469964"
Sep 15 11:34:57.267092 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:57.267033    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:57.267593 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:57.267122    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:57.279504 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:57.279471    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:34:57.349814 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:34:57.349780    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr"
Sep 15 11:35:10.049686 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:10.049645    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:35:23.339327 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.339279    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"]
Sep 15 11:35:23.345717 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.345684    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:23.348414 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.348391    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 15 11:35:23.359449 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.359422    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"]
Sep 15 11:35:23.407360 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.407324    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1781d3ca-d649-4849-b55a-a1bbc9b75f91-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb\" (UID: \"1781d3ca-d649-4849-b55a-a1bbc9b75f91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:23.407544 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.407367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxwdc\" (UniqueName: \"kubernetes.io/projected/1781d3ca-d649-4849-b55a-a1bbc9b75f91-kube-api-access-jxwdc\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb\" (UID: \"1781d3ca-d649-4849-b55a-a1bbc9b75f91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:23.407544 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.407421    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1781d3ca-d649-4849-b55a-a1bbc9b75f91-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb\" (UID: \"1781d3ca-d649-4849-b55a-a1bbc9b75f91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:23.407544 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.407525    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1781d3ca-d649-4849-b55a-a1bbc9b75f91-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb\" (UID: \"1781d3ca-d649-4849-b55a-a1bbc9b75f91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:23.407700 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.407567    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1781d3ca-d649-4849-b55a-a1bbc9b75f91-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb\" (UID: \"1781d3ca-d649-4849-b55a-a1bbc9b75f91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:23.407700 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.407612    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1781d3ca-d649-4849-b55a-a1bbc9b75f91-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb\" (UID: \"1781d3ca-d649-4849-b55a-a1bbc9b75f91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:23.508295 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.508256    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1781d3ca-d649-4849-b55a-a1bbc9b75f91-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb\" (UID: \"1781d3ca-d649-4849-b55a-a1bbc9b75f91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:23.508499 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.508303    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1781d3ca-d649-4849-b55a-a1bbc9b75f91-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb\" (UID: \"1781d3ca-d649-4849-b55a-a1bbc9b75f91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:23.508499 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.508322    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1781d3ca-d649-4849-b55a-a1bbc9b75f91-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb\" (UID: \"1781d3ca-d649-4849-b55a-a1bbc9b75f91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:23.508499 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.508358    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1781d3ca-d649-4849-b55a-a1bbc9b75f91-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb\" (UID: \"1781d3ca-d649-4849-b55a-a1bbc9b75f91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:23.508499 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.508414    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1781d3ca-d649-4849-b55a-a1bbc9b75f91-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb\" (UID: \"1781d3ca-d649-4849-b55a-a1bbc9b75f91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:23.508499 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.508439    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxwdc\" (UniqueName: \"kubernetes.io/projected/1781d3ca-d649-4849-b55a-a1bbc9b75f91-kube-api-access-jxwdc\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb\" (UID: \"1781d3ca-d649-4849-b55a-a1bbc9b75f91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:23.508884 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.508850    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1781d3ca-d649-4849-b55a-a1bbc9b75f91-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb\" (UID: \"1781d3ca-d649-4849-b55a-a1bbc9b75f91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:23.509006 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.508860    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1781d3ca-d649-4849-b55a-a1bbc9b75f91-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb\" (UID: \"1781d3ca-d649-4849-b55a-a1bbc9b75f91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:23.509006 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.508888    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1781d3ca-d649-4849-b55a-a1bbc9b75f91-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb\" (UID: \"1781d3ca-d649-4849-b55a-a1bbc9b75f91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:23.511199 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.511170    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1781d3ca-d649-4849-b55a-a1bbc9b75f91-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb\" (UID: \"1781d3ca-d649-4849-b55a-a1bbc9b75f91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:23.511344 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.511327    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1781d3ca-d649-4849-b55a-a1bbc9b75f91-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb\" (UID: \"1781d3ca-d649-4849-b55a-a1bbc9b75f91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:23.516352 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.516324    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxwdc\" (UniqueName: \"kubernetes.io/projected/1781d3ca-d649-4849-b55a-a1bbc9b75f91-kube-api-access-jxwdc\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb\" (UID: \"1781d3ca-d649-4849-b55a-a1bbc9b75f91\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:23.656165 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.656080    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:23.925382 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:23.925311    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"]
Sep 15 11:35:23.928373 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:35:23.928349    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1781d3ca_d649_4849_b55a_a1bbc9b75f91.slice/crio-f4e8d06db745bb8547761bc27469e9886950b9815e2b1a1739ea85d3548e3bfc WatchSource:0}: Error finding container f4e8d06db745bb8547761bc27469e9886950b9815e2b1a1739ea85d3548e3bfc: Status 404 returned error can't find the container with id f4e8d06db745bb8547761bc27469e9886950b9815e2b1a1739ea85d3548e3bfc
Sep 15 11:35:24.147798 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:24.147768    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:35:24.450334 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:24.450291    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb" event={"ID":"1781d3ca-d649-4849-b55a-a1bbc9b75f91","Type":"ContainerStarted","Data":"a872d915d5342767cbbb6790796d4853b965779a75cde9c071c042a74c455d17"}
Sep 15 11:35:24.450793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:24.450345    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb" event={"ID":"1781d3ca-d649-4849-b55a-a1bbc9b75f91","Type":"ContainerStarted","Data":"f4e8d06db745bb8547761bc27469e9886950b9815e2b1a1739ea85d3548e3bfc"}
Sep 15 11:35:24.468597 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:24.468546    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb" podStartSLOduration=1.285378663 podStartE2EDuration="1.468529817s" totalImagesPullingTime="183.151154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:35:24.467088158 +0000 UTC m=+623.129583355" watchObservedRunningTime="2026-09-15 11:35:24.468529817 +0000 UTC m=+623.131024994"
Sep 15 11:35:33.656234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:33.656191    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:33.658792 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:33.656307    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:33.668684 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:33.668656    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:34.503619 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:34.503590    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb"
Sep 15 11:35:41.643680 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:41.643634    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:35:48.251350 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:35:48.251302    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:36:04.746761 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:04.746726    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:36:12.857271 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:12.857232    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"]
Sep 15 11:36:12.860967 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:12.860939    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:12.863637 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:12.863607    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 15 11:36:12.875328 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:12.875295    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"]
Sep 15 11:36:12.928461 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:12.928423    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f43c5521-3809-429e-9ce0-3dd0c8ef5299-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7spp4\" (UID: \"f43c5521-3809-429e-9ce0-3dd0c8ef5299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:12.928663 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:12.928469    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96rpl\" (UniqueName: \"kubernetes.io/projected/f43c5521-3809-429e-9ce0-3dd0c8ef5299-kube-api-access-96rpl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7spp4\" (UID: \"f43c5521-3809-429e-9ce0-3dd0c8ef5299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:12.928663 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:12.928523    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43c5521-3809-429e-9ce0-3dd0c8ef5299-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7spp4\" (UID: \"f43c5521-3809-429e-9ce0-3dd0c8ef5299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:12.928663 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:12.928558    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43c5521-3809-429e-9ce0-3dd0c8ef5299-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7spp4\" (UID: \"f43c5521-3809-429e-9ce0-3dd0c8ef5299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:12.928663 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:12.928586    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43c5521-3809-429e-9ce0-3dd0c8ef5299-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7spp4\" (UID: \"f43c5521-3809-429e-9ce0-3dd0c8ef5299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:12.928663 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:12.928640    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43c5521-3809-429e-9ce0-3dd0c8ef5299-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7spp4\" (UID: \"f43c5521-3809-429e-9ce0-3dd0c8ef5299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:13.029779 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:13.029744    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43c5521-3809-429e-9ce0-3dd0c8ef5299-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7spp4\" (UID: \"f43c5521-3809-429e-9ce0-3dd0c8ef5299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:13.029967 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:13.029798    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43c5521-3809-429e-9ce0-3dd0c8ef5299-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7spp4\" (UID: \"f43c5521-3809-429e-9ce0-3dd0c8ef5299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:13.029967 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:13.029827    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43c5521-3809-429e-9ce0-3dd0c8ef5299-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7spp4\" (UID: \"f43c5521-3809-429e-9ce0-3dd0c8ef5299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:13.029967 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:13.029871    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43c5521-3809-429e-9ce0-3dd0c8ef5299-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7spp4\" (UID: \"f43c5521-3809-429e-9ce0-3dd0c8ef5299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:13.029967 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:13.029915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f43c5521-3809-429e-9ce0-3dd0c8ef5299-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7spp4\" (UID: \"f43c5521-3809-429e-9ce0-3dd0c8ef5299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:13.029967 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:13.029948    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96rpl\" (UniqueName: \"kubernetes.io/projected/f43c5521-3809-429e-9ce0-3dd0c8ef5299-kube-api-access-96rpl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7spp4\" (UID: \"f43c5521-3809-429e-9ce0-3dd0c8ef5299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:13.030291 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:13.030264    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43c5521-3809-429e-9ce0-3dd0c8ef5299-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7spp4\" (UID: \"f43c5521-3809-429e-9ce0-3dd0c8ef5299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:13.030365 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:13.030314    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f43c5521-3809-429e-9ce0-3dd0c8ef5299-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7spp4\" (UID: \"f43c5521-3809-429e-9ce0-3dd0c8ef5299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:13.030365 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:13.030326    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43c5521-3809-429e-9ce0-3dd0c8ef5299-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7spp4\" (UID: \"f43c5521-3809-429e-9ce0-3dd0c8ef5299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:13.032208 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:13.032187    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43c5521-3809-429e-9ce0-3dd0c8ef5299-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7spp4\" (UID: \"f43c5521-3809-429e-9ce0-3dd0c8ef5299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:13.032481 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:13.032457    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43c5521-3809-429e-9ce0-3dd0c8ef5299-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7spp4\" (UID: \"f43c5521-3809-429e-9ce0-3dd0c8ef5299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:13.039375 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:13.039348    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96rpl\" (UniqueName: \"kubernetes.io/projected/f43c5521-3809-429e-9ce0-3dd0c8ef5299-kube-api-access-96rpl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7spp4\" (UID: \"f43c5521-3809-429e-9ce0-3dd0c8ef5299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:13.171103 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:13.170978    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:13.444334 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:13.444243    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"]
Sep 15 11:36:13.447678 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:36:13.447647    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf43c5521_3809_429e_9ce0_3dd0c8ef5299.slice/crio-78b6626f0866f173f34e60bff85c0ec10e76f57e4d193a47a1efb358f3e8c745 WatchSource:0}: Error finding container 78b6626f0866f173f34e60bff85c0ec10e76f57e4d193a47a1efb358f3e8c745: Status 404 returned error can't find the container with id 78b6626f0866f173f34e60bff85c0ec10e76f57e4d193a47a1efb358f3e8c745
Sep 15 11:36:13.449221 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:13.449199    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 11:36:13.656547 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:13.656513    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4" event={"ID":"f43c5521-3809-429e-9ce0-3dd0c8ef5299","Type":"ContainerStarted","Data":"78b6626f0866f173f34e60bff85c0ec10e76f57e4d193a47a1efb358f3e8c745"}
Sep 15 11:36:14.662424 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:14.662388    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4" event={"ID":"f43c5521-3809-429e-9ce0-3dd0c8ef5299","Type":"ContainerStarted","Data":"ea1a3295ba82a3d317fd9a073335b48056e56ca924db92f711134d831df747a9"}
Sep 15 11:36:14.680188 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:14.680137    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4" podStartSLOduration=2.476037938 podStartE2EDuration="2.680118082s" totalImagesPullingTime="204.080144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:36:14.678717452 +0000 UTC m=+673.341212634" watchObservedRunningTime="2026-09-15 11:36:14.680118082 +0000 UTC m=+673.342613263"
Sep 15 11:36:14.970739 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:14.970656    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:36:23.171202 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:23.171159    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:23.171202 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:23.171212    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:23.183507 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:23.183472    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:36:23.713116 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:36:23.713085    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7spp4"
Sep 15 11:38:04.994425 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:04.994385    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7468-ef7dc6"]
Sep 15 11:38:04.997412 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:04.997394    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7468-ef7dc6"
Sep 15 11:38:04.999910 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:04.999888    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lfxxr\""
Sep 15 11:38:05.004051 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:05.004026    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7468-ef7dc6"]
Sep 15 11:38:05.082108 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:05.082044    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6ctz\" (UniqueName: \"kubernetes.io/projected/da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc-kube-api-access-p6ctz\") pod \"test-curl-7468-ef7dc6\" (UID: \"da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc\") " pod="opendatahub/test-curl-7468-ef7dc6"
Sep 15 11:38:05.183029 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:05.182988    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6ctz\" (UniqueName: \"kubernetes.io/projected/da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc-kube-api-access-p6ctz\") pod \"test-curl-7468-ef7dc6\" (UID: \"da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc\") " pod="opendatahub/test-curl-7468-ef7dc6"
Sep 15 11:38:05.192132 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:05.192104    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6ctz\" (UniqueName: \"kubernetes.io/projected/da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc-kube-api-access-p6ctz\") pod \"test-curl-7468-ef7dc6\" (UID: \"da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc\") " pod="opendatahub/test-curl-7468-ef7dc6"
Sep 15 11:38:05.307309 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:05.307174    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7468-ef7dc6"
Sep 15 11:38:05.576983 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:05.576958    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7468-ef7dc6"]
Sep 15 11:38:05.582299 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:38:05.581342    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda67f5f0_6e5e_4b6a_ae07_55d6ba4099dc.slice/crio-b3aae1e3e4e38fb5946830460eac423310e76d22c09b08e0300f0c05df07c8b4 WatchSource:0}: Error finding container b3aae1e3e4e38fb5946830460eac423310e76d22c09b08e0300f0c05df07c8b4: Status 404 returned error can't find the container with id b3aae1e3e4e38fb5946830460eac423310e76d22c09b08e0300f0c05df07c8b4
Sep 15 11:38:06.128241 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:06.128202    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7468-ef7dc6" event={"ID":"da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc","Type":"ContainerStarted","Data":"b3aae1e3e4e38fb5946830460eac423310e76d22c09b08e0300f0c05df07c8b4"}
Sep 15 11:38:07.354394 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:07.354360    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:38:08.139375 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:08.139333    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7468-ef7dc6" event={"ID":"da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc","Type":"ContainerStarted","Data":"833719c3d3e0e8f79adbf45e268d5ca53e080c31aee59a8c75d7372a87304e1a"}
Sep 15 11:38:08.191949 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:08.191892    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7468-ef7dc6" podStartSLOduration=2.179461191 podStartE2EDuration="4.191875652s" totalImagesPullingTime="2.012414461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:08.190795117 +0000 UTC m=+786.853290299" watchObservedRunningTime="2026-09-15 11:38:08.191875652 +0000 UTC m=+786.854370846"
Sep 15 11:38:08.929347 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:08.929261    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7468-ef7dc6"]
Sep 15 11:38:08.944115 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:08.944082    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7468-ef7dc6"]
Sep 15 11:38:09.077089 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:09.077037    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7468-5961dd"]
Sep 15 11:38:09.077516 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:09.077502    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc" containerName="test-curl-7468-ef7dc6"
Sep 15 11:38:09.077569 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:09.077520    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc" containerName="test-curl-7468-ef7dc6"
Sep 15 11:38:09.077604 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:09.077597    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc" containerName="test-curl-7468-ef7dc6"
Sep 15 11:38:09.080826 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:09.080803    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7468-5961dd"
Sep 15 11:38:09.098249 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:09.098219    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7468-5961dd"]
Sep 15 11:38:09.121384 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:09.121354    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfqtj\" (UniqueName: \"kubernetes.io/projected/4f051200-a20b-4da3-a4e9-479459908bf7-kube-api-access-dfqtj\") pod \"test-curl-7468-5961dd\" (UID: \"4f051200-a20b-4da3-a4e9-479459908bf7\") " pod="opendatahub/test-curl-7468-5961dd"
Sep 15 11:38:09.144285 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:38:09.144249    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7468-ef7dc6" container="test-curl-7468-ef7dc6"
Sep 15 11:38:09.222805 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:09.222712    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfqtj\" (UniqueName: \"kubernetes.io/projected/4f051200-a20b-4da3-a4e9-479459908bf7-kube-api-access-dfqtj\") pod \"test-curl-7468-5961dd\" (UID: \"4f051200-a20b-4da3-a4e9-479459908bf7\") " pod="opendatahub/test-curl-7468-5961dd"
Sep 15 11:38:09.242982 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:09.242954    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfqtj\" (UniqueName: \"kubernetes.io/projected/4f051200-a20b-4da3-a4e9-479459908bf7-kube-api-access-dfqtj\") pod \"test-curl-7468-5961dd\" (UID: \"4f051200-a20b-4da3-a4e9-479459908bf7\") " pod="opendatahub/test-curl-7468-5961dd"
Sep 15 11:38:09.391805 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:09.391772    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7468-5961dd"
Sep 15 11:38:09.663976 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:09.663890    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7468-5961dd"]
Sep 15 11:38:09.667014 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:38:09.666986    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f051200_a20b_4da3_a4e9_479459908bf7.slice/crio-bcf1140052a5cf69d8e7d04fa7ad599c673773fe8f8e02d0024c433038bde6bd WatchSource:0}: Error finding container bcf1140052a5cf69d8e7d04fa7ad599c673773fe8f8e02d0024c433038bde6bd: Status 404 returned error can't find the container with id bcf1140052a5cf69d8e7d04fa7ad599c673773fe8f8e02d0024c433038bde6bd
Sep 15 11:38:09.971042 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:09.970959    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"]
Sep 15 11:38:09.974183 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:09.974164    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"
Sep 15 11:38:09.976500 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:09.976473    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-6e996c-serving-cert\""
Sep 15 11:38:09.976626 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:09.976517    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:38:09.977313 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:09.977291    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 11:38:09.977417 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:09.977314    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bjt69\""
Sep 15 11:38:09.977417 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:09.977371    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 11:38:09.986161 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:09.986096    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"]
Sep 15 11:38:10.029818 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.029778    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad5f1805-33e1-4b10-9b1b-db4099ea5505-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5\" (UID: \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"
Sep 15 11:38:10.030023 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.029831    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad5f1805-33e1-4b10-9b1b-db4099ea5505-metrics-certs\") pod \"maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5\" (UID: \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"
Sep 15 11:38:10.030107 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.030037    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5f1805-33e1-4b10-9b1b-db4099ea5505-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5\" (UID: \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"
Sep 15 11:38:10.030166 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.030119    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq8tk\" (UniqueName: \"kubernetes.io/projected/ad5f1805-33e1-4b10-9b1b-db4099ea5505-kube-api-access-tq8tk\") pod \"maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5\" (UID: \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"
Sep 15 11:38:10.131566 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.131529    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad5f1805-33e1-4b10-9b1b-db4099ea5505-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5\" (UID: \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"
Sep 15 11:38:10.131566 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.131576    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad5f1805-33e1-4b10-9b1b-db4099ea5505-metrics-certs\") pod \"maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5\" (UID: \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"
Sep 15 11:38:10.131779 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.131625    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5f1805-33e1-4b10-9b1b-db4099ea5505-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5\" (UID: \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"
Sep 15 11:38:10.131779 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.131654    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq8tk\" (UniqueName: \"kubernetes.io/projected/ad5f1805-33e1-4b10-9b1b-db4099ea5505-kube-api-access-tq8tk\") pod \"maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5\" (UID: \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"
Sep 15 11:38:10.131779 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:38:10.131690    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-serving-cert: secret "maas-api-e2e-worker-w0-6e996c-serving-cert" not found
Sep 15 11:38:10.131779 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:38:10.131760    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad5f1805-33e1-4b10-9b1b-db4099ea5505-maas-api-tls podName:ad5f1805-33e1-4b10-9b1b-db4099ea5505 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:10.631735763 +0000 UTC m=+789.294230921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ad5f1805-33e1-4b10-9b1b-db4099ea5505-maas-api-tls") pod "maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5" (UID: "ad5f1805-33e1-4b10-9b1b-db4099ea5505") : secret "maas-api-e2e-worker-w0-6e996c-serving-cert" not found
Sep 15 11:38:10.132369 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.132344    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5f1805-33e1-4b10-9b1b-db4099ea5505-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5\" (UID: \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"
Sep 15 11:38:10.134149 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.134127    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad5f1805-33e1-4b10-9b1b-db4099ea5505-metrics-certs\") pod \"maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5\" (UID: \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"
Sep 15 11:38:10.154671 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.154644    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq8tk\" (UniqueName: \"kubernetes.io/projected/ad5f1805-33e1-4b10-9b1b-db4099ea5505-kube-api-access-tq8tk\") pod \"maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5\" (UID: \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"
Sep 15 11:38:10.156618 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.156588    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7468-5961dd" event={"ID":"4f051200-a20b-4da3-a4e9-479459908bf7","Type":"ContainerStarted","Data":"fb8d1583b7bd732b448c75297298fe3b805f581851bee0dc87f29df54cf4070f"}
Sep 15 11:38:10.156751 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.156632    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7468-5961dd" event={"ID":"4f051200-a20b-4da3-a4e9-479459908bf7","Type":"ContainerStarted","Data":"bcf1140052a5cf69d8e7d04fa7ad599c673773fe8f8e02d0024c433038bde6bd"}
Sep 15 11:38:10.156751 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.156628    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7468-ef7dc6" podUID="da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc" containerName="test-curl-7468-ef7dc6" containerID="cri-o://833719c3d3e0e8f79adbf45e268d5ca53e080c31aee59a8c75d7372a87304e1a" gracePeriod=2
Sep 15 11:38:10.172887 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.172845    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7468-5961dd" podStartSLOduration=1.172831437 podStartE2EDuration="1.172831437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:10.171138647 +0000 UTC m=+788.833633828" watchObservedRunningTime="2026-09-15 11:38:10.172831437 +0000 UTC m=+788.835326618"
Sep 15 11:38:10.637135 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.637100    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad5f1805-33e1-4b10-9b1b-db4099ea5505-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5\" (UID: \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"
Sep 15 11:38:10.639353 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.639330    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad5f1805-33e1-4b10-9b1b-db4099ea5505-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5\" (UID: \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"
Sep 15 11:38:10.695381 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.695343    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb"]
Sep 15 11:38:10.698887 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.698865    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb"
Sep 15 11:38:10.701357 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.701323    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 11:38:10.701447 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.701390    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-6e996c-dockercfg-29855\""
Sep 15 11:38:10.702176 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.702160    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 11:38:10.702229 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.702215    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-6e996c\""
Sep 15 11:38:10.717926 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.717899    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb"]
Sep 15 11:38:10.738419 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.738383    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76a47bf-2aea-4ddb-a82a-209bee90a254-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb\" (UID: \"e76a47bf-2aea-4ddb-a82a-209bee90a254\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb"
Sep 15 11:38:10.738571 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.738470    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c46nm\" (UniqueName: \"kubernetes.io/projected/e76a47bf-2aea-4ddb-a82a-209bee90a254-kube-api-access-c46nm\") pod \"payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb\" (UID: \"e76a47bf-2aea-4ddb-a82a-209bee90a254\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb"
Sep 15 11:38:10.802951 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.802907    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7468-5961dd"]
Sep 15 11:38:10.807393 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.807364    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7468-5961dd"]
Sep 15 11:38:10.839335 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.839296    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76a47bf-2aea-4ddb-a82a-209bee90a254-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb\" (UID: \"e76a47bf-2aea-4ddb-a82a-209bee90a254\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb"
Sep 15 11:38:10.839490 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.839408    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c46nm\" (UniqueName: \"kubernetes.io/projected/e76a47bf-2aea-4ddb-a82a-209bee90a254-kube-api-access-c46nm\") pod \"payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb\" (UID: \"e76a47bf-2aea-4ddb-a82a-209bee90a254\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb"
Sep 15 11:38:10.840047 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.840027    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76a47bf-2aea-4ddb-a82a-209bee90a254-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb\" (UID: \"e76a47bf-2aea-4ddb-a82a-209bee90a254\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb"
Sep 15 11:38:10.847272 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.847249    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c46nm\" (UniqueName: \"kubernetes.io/projected/e76a47bf-2aea-4ddb-a82a-209bee90a254-kube-api-access-c46nm\") pod \"payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb\" (UID: \"e76a47bf-2aea-4ddb-a82a-209bee90a254\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb"
Sep 15 11:38:10.886841 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:10.886798    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"
Sep 15 11:38:11.009550 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:11.009508    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb"
Sep 15 11:38:11.160747 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:38:11.160676    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7468-5961dd" container="test-curl-7468-5961dd"
Sep 15 11:38:11.167645 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:11.167621    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"]
Sep 15 11:38:11.169944 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:38:11.169919    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5f1805_33e1_4b10_9b1b_db4099ea5505.slice/crio-98072d0f6d8c2d13d7fce51740478ac84c612a001d7bedbfcc02befaf7969df1 WatchSource:0}: Error finding container 98072d0f6d8c2d13d7fce51740478ac84c612a001d7bedbfcc02befaf7969df1: Status 404 returned error can't find the container with id 98072d0f6d8c2d13d7fce51740478ac84c612a001d7bedbfcc02befaf7969df1
Sep 15 11:38:11.285123 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:11.285097    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb"]
Sep 15 11:38:11.286852 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:38:11.286825    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76a47bf_2aea_4ddb_a82a_209bee90a254.slice/crio-83cd9d904c72d395baa6eb07831c29d278fd5663933caabb8cd89df35ff7637f WatchSource:0}: Error finding container 83cd9d904c72d395baa6eb07831c29d278fd5663933caabb8cd89df35ff7637f: Status 404 returned error can't find the container with id 83cd9d904c72d395baa6eb07831c29d278fd5663933caabb8cd89df35ff7637f
Sep 15 11:38:11.928662 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:11.928622    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6"]
Sep 15 11:38:11.929136 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:11.929117    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f051200-a20b-4da3-a4e9-479459908bf7" containerName="test-curl-7468-5961dd"
Sep 15 11:38:11.929239 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:11.929140    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f051200-a20b-4da3-a4e9-479459908bf7" containerName="test-curl-7468-5961dd"
Sep 15 11:38:11.929305 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:11.929244    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f051200-a20b-4da3-a4e9-479459908bf7" containerName="test-curl-7468-5961dd"
Sep 15 11:38:11.932593 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:11.932571    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6"
Sep 15 11:38:11.935707 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:11.935680    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-b8fbdd\""
Sep 15 11:38:11.936013 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:11.935996    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-b8fbdd-dockercfg-cx6hw\""
Sep 15 11:38:11.946604 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:11.946487    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6"]
Sep 15 11:38:12.063665 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:12.058531    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mkpw\" (UniqueName: \"kubernetes.io/projected/0e09cdb0-f82f-4a5c-a90a-710de63fbacb-kube-api-access-8mkpw\") pod \"payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6\" (UID: \"0e09cdb0-f82f-4a5c-a90a-710de63fbacb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6"
Sep 15 11:38:12.063665 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:12.058593    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e09cdb0-f82f-4a5c-a90a-710de63fbacb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6\" (UID: \"0e09cdb0-f82f-4a5c-a90a-710de63fbacb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6"
Sep 15 11:38:12.159977 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:12.159940    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mkpw\" (UniqueName: \"kubernetes.io/projected/0e09cdb0-f82f-4a5c-a90a-710de63fbacb-kube-api-access-8mkpw\") pod \"payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6\" (UID: \"0e09cdb0-f82f-4a5c-a90a-710de63fbacb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6"
Sep 15 11:38:12.160133 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:12.160009    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e09cdb0-f82f-4a5c-a90a-710de63fbacb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6\" (UID: \"0e09cdb0-f82f-4a5c-a90a-710de63fbacb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6"
Sep 15 11:38:12.160985 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:12.160949    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e09cdb0-f82f-4a5c-a90a-710de63fbacb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6\" (UID: \"0e09cdb0-f82f-4a5c-a90a-710de63fbacb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6"
Sep 15 11:38:12.169975 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:12.169948    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mkpw\" (UniqueName: \"kubernetes.io/projected/0e09cdb0-f82f-4a5c-a90a-710de63fbacb-kube-api-access-8mkpw\") pod \"payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6\" (UID: \"0e09cdb0-f82f-4a5c-a90a-710de63fbacb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6"
Sep 15 11:38:12.176768 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:12.176719    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb" event={"ID":"e76a47bf-2aea-4ddb-a82a-209bee90a254","Type":"ContainerStarted","Data":"83cd9d904c72d395baa6eb07831c29d278fd5663933caabb8cd89df35ff7637f"}
Sep 15 11:38:12.178968 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:12.178900    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7468-5961dd" podUID="4f051200-a20b-4da3-a4e9-479459908bf7" containerName="test-curl-7468-5961dd" containerID="cri-o://fb8d1583b7bd732b448c75297298fe3b805f581851bee0dc87f29df54cf4070f" gracePeriod=2
Sep 15 11:38:12.179250 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:12.179221    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5" event={"ID":"ad5f1805-33e1-4b10-9b1b-db4099ea5505","Type":"ContainerStarted","Data":"98072d0f6d8c2d13d7fce51740478ac84c612a001d7bedbfcc02befaf7969df1"}
Sep 15 11:38:12.254228 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:12.254180    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6"
Sep 15 11:38:12.367397 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:12.367358    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7468-ef7dc6"
Sep 15 11:38:12.464976 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:12.464891    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6ctz\" (UniqueName: \"kubernetes.io/projected/da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc-kube-api-access-p6ctz\") pod \"da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc\" (UID: \"da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc\") "
Sep 15 11:38:12.473520 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:12.473458    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc-kube-api-access-p6ctz" (OuterVolumeSpecName: "kube-api-access-p6ctz") pod "da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc" (UID: "da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc"). InnerVolumeSpecName "kube-api-access-p6ctz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:38:12.566384 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:12.566343    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6ctz\" (UniqueName: \"kubernetes.io/projected/da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc-kube-api-access-p6ctz\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:38:12.579545 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:12.579515    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6"]
Sep 15 11:38:12.582945 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:38:12.582910    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e09cdb0_f82f_4a5c_a90a_710de63fbacb.slice/crio-a60d669a7c39e595576396a998ff2b4860981020f93b7e91c0164973380199a9 WatchSource:0}: Error finding container a60d669a7c39e595576396a998ff2b4860981020f93b7e91c0164973380199a9: Status 404 returned error can't find the container with id a60d669a7c39e595576396a998ff2b4860981020f93b7e91c0164973380199a9
Sep 15 11:38:13.186581 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:13.186545    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6" event={"ID":"0e09cdb0-f82f-4a5c-a90a-710de63fbacb","Type":"ContainerStarted","Data":"a60d669a7c39e595576396a998ff2b4860981020f93b7e91c0164973380199a9"}
Sep 15 11:38:13.188038 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:13.188007    2562 generic.go:356] "Generic (PLEG): container finished" podID="da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc" containerID="833719c3d3e0e8f79adbf45e268d5ca53e080c31aee59a8c75d7372a87304e1a" exitCode=137
Sep 15 11:38:13.188176 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:13.188070    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7468-ef7dc6"
Sep 15 11:38:13.188176 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:13.188082    2562 scope.go:117] "RemoveContainer" containerID="833719c3d3e0e8f79adbf45e268d5ca53e080c31aee59a8c75d7372a87304e1a"
Sep 15 11:38:13.199321 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:38:13.199294    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7468-ef7dc6\" is forbidden: User \"system:node:ip-10-0-135-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-143.ec2.internal' and this object" podUID="da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc" pod="opendatahub/test-curl-7468-ef7dc6"
Sep 15 11:38:13.438617 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:13.438593    2562 scope.go:117] "RemoveContainer" containerID="833719c3d3e0e8f79adbf45e268d5ca53e080c31aee59a8c75d7372a87304e1a"
Sep 15 11:38:13.438981 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:38:13.438965    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"833719c3d3e0e8f79adbf45e268d5ca53e080c31aee59a8c75d7372a87304e1a\": container with ID starting with 833719c3d3e0e8f79adbf45e268d5ca53e080c31aee59a8c75d7372a87304e1a not found: ID does not exist" containerID="833719c3d3e0e8f79adbf45e268d5ca53e080c31aee59a8c75d7372a87304e1a"
Sep 15 11:38:13.439034 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:13.438990    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"833719c3d3e0e8f79adbf45e268d5ca53e080c31aee59a8c75d7372a87304e1a"} err="failed to get container status \"833719c3d3e0e8f79adbf45e268d5ca53e080c31aee59a8c75d7372a87304e1a\": rpc error: code = NotFound desc = could not find container \"833719c3d3e0e8f79adbf45e268d5ca53e080c31aee59a8c75d7372a87304e1a\": container with ID starting with 833719c3d3e0e8f79adbf45e268d5ca53e080c31aee59a8c75d7372a87304e1a not found: ID does not exist"
Sep 15 11:38:13.873394 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:13.873352    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc" path="/var/lib/kubelet/pods/da67f5f0-6e5e-4b6a-ae07-55d6ba4099dc/volumes"
Sep 15 11:38:15.108783 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:15.108758    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7468-5961dd"
Sep 15 11:38:15.191870 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:15.191792    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfqtj\" (UniqueName: \"kubernetes.io/projected/4f051200-a20b-4da3-a4e9-479459908bf7-kube-api-access-dfqtj\") pod \"4f051200-a20b-4da3-a4e9-479459908bf7\" (UID: \"4f051200-a20b-4da3-a4e9-479459908bf7\") "
Sep 15 11:38:15.193872 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:15.193845    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f051200-a20b-4da3-a4e9-479459908bf7-kube-api-access-dfqtj" (OuterVolumeSpecName: "kube-api-access-dfqtj") pod "4f051200-a20b-4da3-a4e9-479459908bf7" (UID: "4f051200-a20b-4da3-a4e9-479459908bf7"). InnerVolumeSpecName "kube-api-access-dfqtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:38:15.204017 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:15.203989    2562 generic.go:356] "Generic (PLEG): container finished" podID="4f051200-a20b-4da3-a4e9-479459908bf7" containerID="fb8d1583b7bd732b448c75297298fe3b805f581851bee0dc87f29df54cf4070f" exitCode=137
Sep 15 11:38:15.204143 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:15.204039    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7468-5961dd"
Sep 15 11:38:15.204143 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:15.204099    2562 scope.go:117] "RemoveContainer" containerID="fb8d1583b7bd732b448c75297298fe3b805f581851bee0dc87f29df54cf4070f"
Sep 15 11:38:15.205840 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:15.205806    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb" event={"ID":"e76a47bf-2aea-4ddb-a82a-209bee90a254","Type":"ContainerStarted","Data":"4542f7a562faed3632c83a96fa87496515fbc1ac0b25c2ef8df8065d3d47569c"}
Sep 15 11:38:15.205958 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:15.205944    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb"
Sep 15 11:38:15.207895 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:15.207868    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5" event={"ID":"ad5f1805-33e1-4b10-9b1b-db4099ea5505","Type":"ContainerStarted","Data":"ba999c4e25221676245c745eb28c9797aa075c7c368bb3bab989883684d77a79"}
Sep 15 11:38:15.208003 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:15.207926    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"
Sep 15 11:38:15.209670 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:15.209646    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6" event={"ID":"0e09cdb0-f82f-4a5c-a90a-710de63fbacb","Type":"ContainerStarted","Data":"7ef16dbbc2063ca09974a0b502d50a170acabaf34ed9ccb7f1f4b812c55b8b35"}
Sep 15 11:38:15.209790 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:15.209698    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6"
Sep 15 11:38:15.213980 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:15.213962    2562 scope.go:117] "RemoveContainer" containerID="fb8d1583b7bd732b448c75297298fe3b805f581851bee0dc87f29df54cf4070f"
Sep 15 11:38:15.214276 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:38:15.214256    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb8d1583b7bd732b448c75297298fe3b805f581851bee0dc87f29df54cf4070f\": container with ID starting with fb8d1583b7bd732b448c75297298fe3b805f581851bee0dc87f29df54cf4070f not found: ID does not exist" containerID="fb8d1583b7bd732b448c75297298fe3b805f581851bee0dc87f29df54cf4070f"
Sep 15 11:38:15.214357 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:15.214282    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb8d1583b7bd732b448c75297298fe3b805f581851bee0dc87f29df54cf4070f"} err="failed to get container status \"fb8d1583b7bd732b448c75297298fe3b805f581851bee0dc87f29df54cf4070f\": rpc error: code = NotFound desc = could not find container \"fb8d1583b7bd732b448c75297298fe3b805f581851bee0dc87f29df54cf4070f\": container with ID starting with fb8d1583b7bd732b448c75297298fe3b805f581851bee0dc87f29df54cf4070f not found: ID does not exist"
Sep 15 11:38:15.287017 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:15.286956    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb" podStartSLOduration=1.59347188 podStartE2EDuration="5.28693453s" totalImagesPullingTime="3.69346265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:15.277104123 +0000 UTC m=+793.939599322" watchObservedRunningTime="2026-09-15 11:38:15.28693453 +0000 UTC m=+793.949429712"
Sep 15 11:38:15.293316 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:15.292933    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfqtj\" (UniqueName: \"kubernetes.io/projected/4f051200-a20b-4da3-a4e9-479459908bf7-kube-api-access-dfqtj\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:38:15.365721 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:15.365663    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6" podStartSLOduration=1.9661337190000001 podStartE2EDuration="4.365647343s" totalImagesPullingTime="2.399513624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:15.364962094 +0000 UTC m=+794.027457305" watchObservedRunningTime="2026-09-15 11:38:15.365647343 +0000 UTC m=+794.028142547"
Sep 15 11:38:15.471969 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:15.471854    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5" podStartSLOduration=2.660543476 podStartE2EDuration="6.471836259s" totalImagesPullingTime="3.811292783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:15.470853035 +0000 UTC m=+794.133348227" watchObservedRunningTime="2026-09-15 11:38:15.471836259 +0000 UTC m=+794.134331466"
Sep 15 11:38:15.473739 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:38:15.473711    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7468-5961dd\" is forbidden: User \"system:node:ip-10-0-135-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-143.ec2.internal' and this object" podUID="4f051200-a20b-4da3-a4e9-479459908bf7" pod="opendatahub/test-curl-7468-5961dd"
Sep 15 11:38:15.873997 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:15.873964    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f051200-a20b-4da3-a4e9-479459908bf7" path="/var/lib/kubelet/pods/4f051200-a20b-4da3-a4e9-479459908bf7/volumes"
Sep 15 11:38:16.683906 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:16.683870    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"]
Sep 15 11:38:16.689163 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:16.689142    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"
Sep 15 11:38:16.691270 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:16.691247    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-4d6dbb-serving-cert\""
Sep 15 11:38:16.701175 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:16.701149    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"]
Sep 15 11:38:16.808546 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:16.808507    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zv46\" (UniqueName: \"kubernetes.io/projected/62710c30-a12e-467b-bdca-d4490c646683-kube-api-access-2zv46\") pod \"maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7\" (UID: \"62710c30-a12e-467b-bdca-d4490c646683\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"
Sep 15 11:38:16.808546 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:16.808548    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62710c30-a12e-467b-bdca-d4490c646683-maas-api-tls\") pod \"maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7\" (UID: \"62710c30-a12e-467b-bdca-d4490c646683\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"
Sep 15 11:38:16.808748 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:16.808609    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62710c30-a12e-467b-bdca-d4490c646683-metrics-certs\") pod \"maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7\" (UID: \"62710c30-a12e-467b-bdca-d4490c646683\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"
Sep 15 11:38:16.808748 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:16.808641    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62710c30-a12e-467b-bdca-d4490c646683-openshift-service-ca\") pod \"maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7\" (UID: \"62710c30-a12e-467b-bdca-d4490c646683\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"
Sep 15 11:38:16.909579 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:16.909538    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62710c30-a12e-467b-bdca-d4490c646683-metrics-certs\") pod \"maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7\" (UID: \"62710c30-a12e-467b-bdca-d4490c646683\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"
Sep 15 11:38:16.909736 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:16.909602    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62710c30-a12e-467b-bdca-d4490c646683-openshift-service-ca\") pod \"maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7\" (UID: \"62710c30-a12e-467b-bdca-d4490c646683\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"
Sep 15 11:38:16.909736 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:16.909696    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zv46\" (UniqueName: \"kubernetes.io/projected/62710c30-a12e-467b-bdca-d4490c646683-kube-api-access-2zv46\") pod \"maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7\" (UID: \"62710c30-a12e-467b-bdca-d4490c646683\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"
Sep 15 11:38:16.909736 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:16.909721    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62710c30-a12e-467b-bdca-d4490c646683-maas-api-tls\") pod \"maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7\" (UID: \"62710c30-a12e-467b-bdca-d4490c646683\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"
Sep 15 11:38:16.909847 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:38:16.909838    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-serving-cert: secret "maas-api-e2e-autores-4d6dbb-serving-cert" not found
Sep 15 11:38:16.909915 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:38:16.909904    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62710c30-a12e-467b-bdca-d4490c646683-maas-api-tls podName:62710c30-a12e-467b-bdca-d4490c646683 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:17.409883683 +0000 UTC m=+796.072378849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/62710c30-a12e-467b-bdca-d4490c646683-maas-api-tls") pod "maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7" (UID: "62710c30-a12e-467b-bdca-d4490c646683") : secret "maas-api-e2e-autores-4d6dbb-serving-cert" not found
Sep 15 11:38:16.910398 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:16.910374    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62710c30-a12e-467b-bdca-d4490c646683-openshift-service-ca\") pod \"maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7\" (UID: \"62710c30-a12e-467b-bdca-d4490c646683\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"
Sep 15 11:38:16.912158 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:16.912136    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62710c30-a12e-467b-bdca-d4490c646683-metrics-certs\") pod \"maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7\" (UID: \"62710c30-a12e-467b-bdca-d4490c646683\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"
Sep 15 11:38:16.922534 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:16.922498    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zv46\" (UniqueName: \"kubernetes.io/projected/62710c30-a12e-467b-bdca-d4490c646683-kube-api-access-2zv46\") pod \"maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7\" (UID: \"62710c30-a12e-467b-bdca-d4490c646683\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"
Sep 15 11:38:17.414503 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:17.414463    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62710c30-a12e-467b-bdca-d4490c646683-maas-api-tls\") pod \"maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7\" (UID: \"62710c30-a12e-467b-bdca-d4490c646683\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"
Sep 15 11:38:17.416897 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:17.416870    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62710c30-a12e-467b-bdca-d4490c646683-maas-api-tls\") pod \"maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7\" (UID: \"62710c30-a12e-467b-bdca-d4490c646683\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"
Sep 15 11:38:17.603245 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:17.603207    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"
Sep 15 11:38:17.890313 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:17.890268    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"]
Sep 15 11:38:17.894928 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:38:17.894899    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62710c30_a12e_467b_bdca_d4490c646683.slice/crio-626aefa1e3e2158f51d3853890b4d503a73661f755c58788034197c8e451a974 WatchSource:0}: Error finding container 626aefa1e3e2158f51d3853890b4d503a73661f755c58788034197c8e451a974: Status 404 returned error can't find the container with id 626aefa1e3e2158f51d3853890b4d503a73661f755c58788034197c8e451a974
Sep 15 11:38:18.026264 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:18.026223    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl"]
Sep 15 11:38:18.032002 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:18.031978    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl"
Sep 15 11:38:18.039904 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:18.039882    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-4d6dbb\""
Sep 15 11:38:18.040207 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:18.040192    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-4d6dbb-dockercfg-579gx\""
Sep 15 11:38:18.059789 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:18.059762    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl"]
Sep 15 11:38:18.121534 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:18.121498    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ff4115-8602-42ab-804a-3098defbc45c-plugins-config-volume\") pod \"payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl\" (UID: \"f8ff4115-8602-42ab-804a-3098defbc45c\") " pod="openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl"
Sep 15 11:38:18.121683 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:18.121556    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx6xd\" (UniqueName: \"kubernetes.io/projected/f8ff4115-8602-42ab-804a-3098defbc45c-kube-api-access-rx6xd\") pod \"payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl\" (UID: \"f8ff4115-8602-42ab-804a-3098defbc45c\") " pod="openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl"
Sep 15 11:38:18.222147 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:18.222034    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ff4115-8602-42ab-804a-3098defbc45c-plugins-config-volume\") pod \"payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl\" (UID: \"f8ff4115-8602-42ab-804a-3098defbc45c\") " pod="openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl"
Sep 15 11:38:18.222147 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:18.222131    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx6xd\" (UniqueName: \"kubernetes.io/projected/f8ff4115-8602-42ab-804a-3098defbc45c-kube-api-access-rx6xd\") pod \"payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl\" (UID: \"f8ff4115-8602-42ab-804a-3098defbc45c\") " pod="openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl"
Sep 15 11:38:18.222771 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:18.222747    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ff4115-8602-42ab-804a-3098defbc45c-plugins-config-volume\") pod \"payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl\" (UID: \"f8ff4115-8602-42ab-804a-3098defbc45c\") " pod="openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl"
Sep 15 11:38:18.224803 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:18.224777    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7" event={"ID":"62710c30-a12e-467b-bdca-d4490c646683","Type":"ContainerStarted","Data":"626aefa1e3e2158f51d3853890b4d503a73661f755c58788034197c8e451a974"}
Sep 15 11:38:18.234991 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:18.234969    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx6xd\" (UniqueName: \"kubernetes.io/projected/f8ff4115-8602-42ab-804a-3098defbc45c-kube-api-access-rx6xd\") pod \"payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl\" (UID: \"f8ff4115-8602-42ab-804a-3098defbc45c\") " pod="openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl"
Sep 15 11:38:18.342073 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:18.342033    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl"
Sep 15 11:38:18.654544 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:18.654510    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl"]
Sep 15 11:38:18.657385 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:38:18.657359    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ff4115_8602_42ab_804a_3098defbc45c.slice/crio-b041662a8623fb9f9763824c79b4690ee2c62f292440570909014336bddd88bd WatchSource:0}: Error finding container b041662a8623fb9f9763824c79b4690ee2c62f292440570909014336bddd88bd: Status 404 returned error can't find the container with id b041662a8623fb9f9763824c79b4690ee2c62f292440570909014336bddd88bd
Sep 15 11:38:19.231251 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:19.231215    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7" event={"ID":"62710c30-a12e-467b-bdca-d4490c646683","Type":"ContainerStarted","Data":"4f0249506408b20a6f11921b707fa66a780769d9438ba9e585811526c1a1dd64"}
Sep 15 11:38:19.231724 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:19.231320    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"
Sep 15 11:38:19.232940 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:19.232910    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl" event={"ID":"f8ff4115-8602-42ab-804a-3098defbc45c","Type":"ContainerStarted","Data":"b240bb6c7683fe51bd6b01506330b7cc4fc66b7dfb716883cb1310acb8b59352"}
Sep 15 11:38:19.233072 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:19.232958    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl" event={"ID":"f8ff4115-8602-42ab-804a-3098defbc45c","Type":"ContainerStarted","Data":"b041662a8623fb9f9763824c79b4690ee2c62f292440570909014336bddd88bd"}
Sep 15 11:38:19.233072 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:19.232977    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl"
Sep 15 11:38:19.269660 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:19.269611    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7" podStartSLOduration=2.9162377 podStartE2EDuration="3.26959312s" totalImagesPullingTime="353.35542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:19.266487433 +0000 UTC m=+797.928982614" watchObservedRunningTime="2026-09-15 11:38:19.26959312 +0000 UTC m=+797.932088303"
Sep 15 11:38:19.303106 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:19.303044    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl" podStartSLOduration=2.303030216 podStartE2EDuration="2.303030216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:19.300909377 +0000 UTC m=+797.963404577" watchObservedRunningTime="2026-09-15 11:38:19.303030216 +0000 UTC m=+797.965525397"
Sep 15 11:38:21.220423 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:21.220395    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"
Sep 15 11:38:25.243170 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:25.243096    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"
Sep 15 11:38:26.216580 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:26.216546    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb"
Sep 15 11:38:26.216959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:26.216939    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6"
Sep 15 11:38:28.161832 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:28.161795    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:38:30.239921 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:30.239885    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl"
Sep 15 11:38:30.804136 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:30.804103    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"]
Sep 15 11:38:30.807663 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:30.807644    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"
Sep 15 11:38:30.810319 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:30.810133    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-78522e-serving-cert\""
Sep 15 11:38:30.822168 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:30.822143    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"]
Sep 15 11:38:30.932703 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:30.932666    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smdv9\" (UniqueName: \"kubernetes.io/projected/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-kube-api-access-smdv9\") pod \"maas-api-e2e-inf-b-78522e-66f659895d-drpkg\" (UID: \"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"
Sep 15 11:38:30.932703 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:30.932708    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-maas-api-tls\") pod \"maas-api-e2e-inf-b-78522e-66f659895d-drpkg\" (UID: \"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"
Sep 15 11:38:30.932921 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:30.932739    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-metrics-certs\") pod \"maas-api-e2e-inf-b-78522e-66f659895d-drpkg\" (UID: \"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"
Sep 15 11:38:30.932921 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:30.932767    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-78522e-66f659895d-drpkg\" (UID: \"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"
Sep 15 11:38:31.033698 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.033456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smdv9\" (UniqueName: \"kubernetes.io/projected/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-kube-api-access-smdv9\") pod \"maas-api-e2e-inf-b-78522e-66f659895d-drpkg\" (UID: \"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"
Sep 15 11:38:31.033698 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.033677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-maas-api-tls\") pod \"maas-api-e2e-inf-b-78522e-66f659895d-drpkg\" (UID: \"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"
Sep 15 11:38:31.033956 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.033743    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-metrics-certs\") pod \"maas-api-e2e-inf-b-78522e-66f659895d-drpkg\" (UID: \"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"
Sep 15 11:38:31.033956 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.033784    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-78522e-66f659895d-drpkg\" (UID: \"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"
Sep 15 11:38:31.034391 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.034365    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-78522e-66f659895d-drpkg\" (UID: \"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"
Sep 15 11:38:31.036067 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.036030    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-maas-api-tls\") pod \"maas-api-e2e-inf-b-78522e-66f659895d-drpkg\" (UID: \"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"
Sep 15 11:38:31.036165 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.036100    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-metrics-certs\") pod \"maas-api-e2e-inf-b-78522e-66f659895d-drpkg\" (UID: \"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"
Sep 15 11:38:31.042603 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.042578    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smdv9\" (UniqueName: \"kubernetes.io/projected/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-kube-api-access-smdv9\") pod \"maas-api-e2e-inf-b-78522e-66f659895d-drpkg\" (UID: \"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"
Sep 15 11:38:31.120373 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.120343    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"
Sep 15 11:38:31.395712 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.395653    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"]
Sep 15 11:38:31.397988 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:38:31.397947    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3b74e3_6049_4f0a_a7eb_6eeddd98ea9c.slice/crio-94f6a91754077dddd4b019e93db844cb6252c2ff23cf4b4355bb3d08e9ff94fc WatchSource:0}: Error finding container 94f6a91754077dddd4b019e93db844cb6252c2ff23cf4b4355bb3d08e9ff94fc: Status 404 returned error can't find the container with id 94f6a91754077dddd4b019e93db844cb6252c2ff23cf4b4355bb3d08e9ff94fc
Sep 15 11:38:31.727454 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.727371    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5"]
Sep 15 11:38:31.731592 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.731576    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5"
Sep 15 11:38:31.733536 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.733515    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-78522e\""
Sep 15 11:38:31.733616 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.733599    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-78522e-dockercfg-8m9q7\""
Sep 15 11:38:31.740410 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.740389    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5"]
Sep 15 11:38:31.842770 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.842738    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3059f98-22e7-464b-8c09-a2c2d98d056f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5\" (UID: \"c3059f98-22e7-464b-8c09-a2c2d98d056f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5"
Sep 15 11:38:31.842872 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.842857    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsjnr\" (UniqueName: \"kubernetes.io/projected/c3059f98-22e7-464b-8c09-a2c2d98d056f-kube-api-access-qsjnr\") pod \"payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5\" (UID: \"c3059f98-22e7-464b-8c09-a2c2d98d056f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5"
Sep 15 11:38:31.943445 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.943405    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3059f98-22e7-464b-8c09-a2c2d98d056f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5\" (UID: \"c3059f98-22e7-464b-8c09-a2c2d98d056f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5"
Sep 15 11:38:31.943632 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.943492    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsjnr\" (UniqueName: \"kubernetes.io/projected/c3059f98-22e7-464b-8c09-a2c2d98d056f-kube-api-access-qsjnr\") pod \"payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5\" (UID: \"c3059f98-22e7-464b-8c09-a2c2d98d056f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5"
Sep 15 11:38:31.944151 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.944127    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3059f98-22e7-464b-8c09-a2c2d98d056f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5\" (UID: \"c3059f98-22e7-464b-8c09-a2c2d98d056f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5"
Sep 15 11:38:31.952667 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:31.952647    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsjnr\" (UniqueName: \"kubernetes.io/projected/c3059f98-22e7-464b-8c09-a2c2d98d056f-kube-api-access-qsjnr\") pod \"payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5\" (UID: \"c3059f98-22e7-464b-8c09-a2c2d98d056f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5"
Sep 15 11:38:32.043628 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:32.043553    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5"
Sep 15 11:38:32.298520 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:32.298431    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg" event={"ID":"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c","Type":"ContainerStarted","Data":"cc8d45d9e4592edb43f030302415bd59ddf408cfcf685cc0ffedea39b53bfcf2"}
Sep 15 11:38:32.298520 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:32.298512    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg" event={"ID":"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c","Type":"ContainerStarted","Data":"94f6a91754077dddd4b019e93db844cb6252c2ff23cf4b4355bb3d08e9ff94fc"}
Sep 15 11:38:32.298712 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:32.298548    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"
Sep 15 11:38:32.332618 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:32.332419    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg" podStartSLOduration=1.946088918 podStartE2EDuration="2.332402021s" totalImagesPullingTime="386.313103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:32.330870634 +0000 UTC m=+810.993365837" watchObservedRunningTime="2026-09-15 11:38:32.332402021 +0000 UTC m=+810.994897201"
Sep 15 11:38:32.332881 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:32.332864    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5"]
Sep 15 11:38:32.335380 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:38:32.335342    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3059f98_22e7_464b_8c09_a2c2d98d056f.slice/crio-f72ac2ae7ad47361c9b0c4694377f87ccb9294075477cb8ea0620815f7edc6ca WatchSource:0}: Error finding container f72ac2ae7ad47361c9b0c4694377f87ccb9294075477cb8ea0620815f7edc6ca: Status 404 returned error can't find the container with id f72ac2ae7ad47361c9b0c4694377f87ccb9294075477cb8ea0620815f7edc6ca
Sep 15 11:38:33.304315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:33.304272    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5" event={"ID":"c3059f98-22e7-464b-8c09-a2c2d98d056f","Type":"ContainerStarted","Data":"59e04cf708360c463d48845821dad2b1933b892aac0a2c3d141bbc6e51829ad6"}
Sep 15 11:38:33.304758 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:33.304330    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5" event={"ID":"c3059f98-22e7-464b-8c09-a2c2d98d056f","Type":"ContainerStarted","Data":"f72ac2ae7ad47361c9b0c4694377f87ccb9294075477cb8ea0620815f7edc6ca"}
Sep 15 11:38:33.304758 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:33.304635    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5"
Sep 15 11:38:33.326111 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:33.326046    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5" podStartSLOduration=2.32603077 podStartE2EDuration="2.32603077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:33.32484867 +0000 UTC m=+811.987343881" watchObservedRunningTime="2026-09-15 11:38:33.32603077 +0000 UTC m=+811.988525953"
Sep 15 11:38:38.310132 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:38.310104    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"
Sep 15 11:38:44.311049 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:44.311013    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5"
Sep 15 11:38:49.665267 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:49.665226    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:38:58.526658 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:58.526618    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs"]
Sep 15 11:38:58.530101 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:58.530051    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs"
Sep 15 11:38:58.532617 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:58.532596    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-07a6c076-dockercfg-bm9j6\""
Sep 15 11:38:58.532898 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:58.532882    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-07a6c076\""
Sep 15 11:38:58.546954 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:58.546927    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs"]
Sep 15 11:38:58.548291 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:58.548268    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed617b9c-c4f8-49cc-b431-6d97fa8a0083-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs\" (UID: \"ed617b9c-c4f8-49cc-b431-6d97fa8a0083\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs"
Sep 15 11:38:58.548402 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:58.548383    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtwv8\" (UniqueName: \"kubernetes.io/projected/ed617b9c-c4f8-49cc-b431-6d97fa8a0083-kube-api-access-jtwv8\") pod \"payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs\" (UID: \"ed617b9c-c4f8-49cc-b431-6d97fa8a0083\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs"
Sep 15 11:38:58.649174 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:58.649133    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtwv8\" (UniqueName: \"kubernetes.io/projected/ed617b9c-c4f8-49cc-b431-6d97fa8a0083-kube-api-access-jtwv8\") pod \"payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs\" (UID: \"ed617b9c-c4f8-49cc-b431-6d97fa8a0083\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs"
Sep 15 11:38:58.649351 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:58.649191    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed617b9c-c4f8-49cc-b431-6d97fa8a0083-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs\" (UID: \"ed617b9c-c4f8-49cc-b431-6d97fa8a0083\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs"
Sep 15 11:38:58.649831 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:58.649813    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed617b9c-c4f8-49cc-b431-6d97fa8a0083-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs\" (UID: \"ed617b9c-c4f8-49cc-b431-6d97fa8a0083\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs"
Sep 15 11:38:58.663300 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:58.663265    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtwv8\" (UniqueName: \"kubernetes.io/projected/ed617b9c-c4f8-49cc-b431-6d97fa8a0083-kube-api-access-jtwv8\") pod \"payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs\" (UID: \"ed617b9c-c4f8-49cc-b431-6d97fa8a0083\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs"
Sep 15 11:38:58.839834 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:58.839795    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs"
Sep 15 11:38:59.113636 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.113610    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs"]
Sep 15 11:38:59.335844 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.335803    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"]
Sep 15 11:38:59.339344 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.339320    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:38:59.342220 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.342196    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6e996c\"/\"kube-root-ca.crt\""
Sep 15 11:38:59.342365 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.342348    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6e996c\"/\"openshift-service-ca.crt\""
Sep 15 11:38:59.342437 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.342377    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6e996c\"/\"simulator-w0-6e996c-kserve-self-signed-certs\""
Sep 15 11:38:59.342492 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.342436    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6e996c\"/\"default-dockercfg-hqdd9\""
Sep 15 11:38:59.350174 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.350155    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"]
Sep 15 11:38:59.355748 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.355723    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-model-cache\") pod \"simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") " pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:38:59.355848 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.355816    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-dshm\") pod \"simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") " pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:38:59.355918 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.355869    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-home\") pod \"simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") " pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:38:59.355918 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.355904    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-tmp-dir\") pod \"simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") " pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:38:59.356011 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.355992    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpnrr\" (UniqueName: \"kubernetes.io/projected/3a04793a-5f2b-4920-8acb-dd0639c1b04f-kube-api-access-gpnrr\") pod \"simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") " pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:38:59.356078 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.356046    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a04793a-5f2b-4920-8acb-dd0639c1b04f-tls-certs\") pod \"simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") " pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:38:59.423799 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.423714    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs" event={"ID":"ed617b9c-c4f8-49cc-b431-6d97fa8a0083","Type":"ContainerStarted","Data":"7c6107b28c7e0b4cfc7636960b0f404f61bee903a6b94b11530c08954df599c2"}
Sep 15 11:38:59.423799 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.423763    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs" event={"ID":"ed617b9c-c4f8-49cc-b431-6d97fa8a0083","Type":"ContainerStarted","Data":"c2b6d10262817b88149d711af4ceafe0df524c7eadacc4fe50bcad179ed8bb29"}
Sep 15 11:38:59.424051 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.423860    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs"
Sep 15 11:38:59.441749 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.441675    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs" podStartSLOduration=1.441653175 podStartE2EDuration="1.441653175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:59.439162518 +0000 UTC m=+838.101657719" watchObservedRunningTime="2026-09-15 11:38:59.441653175 +0000 UTC m=+838.104148361"
Sep 15 11:38:59.456694 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.456658    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-dshm\") pod \"simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") " pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:38:59.456864 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.456704    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-home\") pod \"simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") " pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:38:59.456864 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.456821    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-tmp-dir\") pod \"simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") " pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:38:59.457002 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.456913    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpnrr\" (UniqueName: \"kubernetes.io/projected/3a04793a-5f2b-4920-8acb-dd0639c1b04f-kube-api-access-gpnrr\") pod \"simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") " pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:38:59.457002 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.456955    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a04793a-5f2b-4920-8acb-dd0639c1b04f-tls-certs\") pod \"simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") " pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:38:59.457144 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.457126    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-home\") pod \"simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") " pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:38:59.457317 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.457231    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-tmp-dir\") pod \"simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") " pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:38:59.457317 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.457236    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-model-cache\") pod \"simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") " pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:38:59.457547 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.457524    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-model-cache\") pod \"simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") " pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:38:59.459149 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.459131    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-dshm\") pod \"simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") " pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:38:59.459415 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.459396    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a04793a-5f2b-4920-8acb-dd0639c1b04f-tls-certs\") pod \"simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") " pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:38:59.465115 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.465091    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpnrr\" (UniqueName: \"kubernetes.io/projected/3a04793a-5f2b-4920-8acb-dd0639c1b04f-kube-api-access-gpnrr\") pod \"simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") " pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:38:59.649973 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.649931    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:38:59.930236 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:38:59.930027    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"]
Sep 15 11:38:59.932694 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:38:59.932664    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a04793a_5f2b_4920_8acb_dd0639c1b04f.slice/crio-2a8f83a08cb6d0efec34a97aa422275c51a0e1bdb98ed8607a2e6b67e25c67ac WatchSource:0}: Error finding container 2a8f83a08cb6d0efec34a97aa422275c51a0e1bdb98ed8607a2e6b67e25c67ac: Status 404 returned error can't find the container with id 2a8f83a08cb6d0efec34a97aa422275c51a0e1bdb98ed8607a2e6b67e25c67ac
Sep 15 11:39:00.431141 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:00.431094    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67" event={"ID":"3a04793a-5f2b-4920-8acb-dd0639c1b04f","Type":"ContainerStarted","Data":"a83535ed9a0786e8678faa71656b701a11199d7f38c996cc444f0c84c86ff252"}
Sep 15 11:39:00.431141 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:00.431147    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67" event={"ID":"3a04793a-5f2b-4920-8acb-dd0639c1b04f","Type":"ContainerStarted","Data":"2a8f83a08cb6d0efec34a97aa422275c51a0e1bdb98ed8607a2e6b67e25c67ac"}
Sep 15 11:39:00.453832 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:00.453783    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67" podStartSLOduration=1.453766401 podStartE2EDuration="1.453766401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:00.452571728 +0000 UTC m=+839.115066908" watchObservedRunningTime="2026-09-15 11:39:00.453766401 +0000 UTC m=+839.116261581"
Sep 15 11:39:07.863856 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:07.863818    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:39:09.651009 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:09.650968    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:39:09.651432 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:09.651068    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:39:09.663917 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:09.663879    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:39:10.433916 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:10.433883    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs"
Sep 15 11:39:10.492855 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:10.492823    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:39:26.933579 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:26.933538    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs"]
Sep 15 11:39:26.934192 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:26.933802    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs" podUID="ed617b9c-c4f8-49cc-b431-6d97fa8a0083" containerName="payload-pre-processing" containerID="cri-o://7c6107b28c7e0b4cfc7636960b0f404f61bee903a6b94b11530c08954df599c2" gracePeriod=30
Sep 15 11:39:27.191117 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:27.191043    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs"
Sep 15 11:39:27.311192 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:27.311160    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtwv8\" (UniqueName: \"kubernetes.io/projected/ed617b9c-c4f8-49cc-b431-6d97fa8a0083-kube-api-access-jtwv8\") pod \"ed617b9c-c4f8-49cc-b431-6d97fa8a0083\" (UID: \"ed617b9c-c4f8-49cc-b431-6d97fa8a0083\") "
Sep 15 11:39:27.311392 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:27.311264    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed617b9c-c4f8-49cc-b431-6d97fa8a0083-plugins-config-volume\") pod \"ed617b9c-c4f8-49cc-b431-6d97fa8a0083\" (UID: \"ed617b9c-c4f8-49cc-b431-6d97fa8a0083\") "
Sep 15 11:39:27.311657 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:27.311635    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed617b9c-c4f8-49cc-b431-6d97fa8a0083-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed617b9c-c4f8-49cc-b431-6d97fa8a0083" (UID: "ed617b9c-c4f8-49cc-b431-6d97fa8a0083"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:39:27.313598 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:27.313568    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed617b9c-c4f8-49cc-b431-6d97fa8a0083-kube-api-access-jtwv8" (OuterVolumeSpecName: "kube-api-access-jtwv8") pod "ed617b9c-c4f8-49cc-b431-6d97fa8a0083" (UID: "ed617b9c-c4f8-49cc-b431-6d97fa8a0083"). InnerVolumeSpecName "kube-api-access-jtwv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:39:27.412366 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:27.412326    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtwv8\" (UniqueName: \"kubernetes.io/projected/ed617b9c-c4f8-49cc-b431-6d97fa8a0083-kube-api-access-jtwv8\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:39:27.412366 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:27.412358    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed617b9c-c4f8-49cc-b431-6d97fa8a0083-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:39:27.557772 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:27.557684    2562 generic.go:356] "Generic (PLEG): container finished" podID="ed617b9c-c4f8-49cc-b431-6d97fa8a0083" containerID="7c6107b28c7e0b4cfc7636960b0f404f61bee903a6b94b11530c08954df599c2" exitCode=0
Sep 15 11:39:27.557772 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:27.557732    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs"
Sep 15 11:39:27.557945 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:27.557767    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs" event={"ID":"ed617b9c-c4f8-49cc-b431-6d97fa8a0083","Type":"ContainerDied","Data":"7c6107b28c7e0b4cfc7636960b0f404f61bee903a6b94b11530c08954df599c2"}
Sep 15 11:39:27.557945 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:27.557818    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs" event={"ID":"ed617b9c-c4f8-49cc-b431-6d97fa8a0083","Type":"ContainerDied","Data":"c2b6d10262817b88149d711af4ceafe0df524c7eadacc4fe50bcad179ed8bb29"}
Sep 15 11:39:27.557945 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:27.557839    2562 scope.go:117] "RemoveContainer" containerID="7c6107b28c7e0b4cfc7636960b0f404f61bee903a6b94b11530c08954df599c2"
Sep 15 11:39:27.566591 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:27.566574    2562 scope.go:117] "RemoveContainer" containerID="7c6107b28c7e0b4cfc7636960b0f404f61bee903a6b94b11530c08954df599c2"
Sep 15 11:39:27.566874 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:39:27.566854    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c6107b28c7e0b4cfc7636960b0f404f61bee903a6b94b11530c08954df599c2\": container with ID starting with 7c6107b28c7e0b4cfc7636960b0f404f61bee903a6b94b11530c08954df599c2 not found: ID does not exist" containerID="7c6107b28c7e0b4cfc7636960b0f404f61bee903a6b94b11530c08954df599c2"
Sep 15 11:39:27.566926 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:27.566884    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c6107b28c7e0b4cfc7636960b0f404f61bee903a6b94b11530c08954df599c2"} err="failed to get container status \"7c6107b28c7e0b4cfc7636960b0f404f61bee903a6b94b11530c08954df599c2\": rpc error: code = NotFound desc = could not find container \"7c6107b28c7e0b4cfc7636960b0f404f61bee903a6b94b11530c08954df599c2\": container with ID starting with 7c6107b28c7e0b4cfc7636960b0f404f61bee903a6b94b11530c08954df599c2 not found: ID does not exist"
Sep 15 11:39:27.577809 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:27.577783    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs"]
Sep 15 11:39:27.582108 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:27.582085    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-07a6c076-5f9fcb459d-ksczs"]
Sep 15 11:39:27.873792 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:27.873756    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed617b9c-c4f8-49cc-b431-6d97fa8a0083" path="/var/lib/kubelet/pods/ed617b9c-c4f8-49cc-b431-6d97fa8a0083/volumes"
Sep 15 11:39:33.779917 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:33.779879    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf"]
Sep 15 11:39:33.780330 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:33.780278    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed617b9c-c4f8-49cc-b431-6d97fa8a0083" containerName="payload-pre-processing"
Sep 15 11:39:33.780330 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:33.780299    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed617b9c-c4f8-49cc-b431-6d97fa8a0083" containerName="payload-pre-processing"
Sep 15 11:39:33.780402 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:33.780373    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed617b9c-c4f8-49cc-b431-6d97fa8a0083" containerName="payload-pre-processing"
Sep 15 11:39:33.782392 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:33.782375    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf"
Sep 15 11:39:33.785680 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:33.785657    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9b1cbda7-dockercfg-prg2s\""
Sep 15 11:39:33.785853 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:33.785737    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9b1cbda7\""
Sep 15 11:39:33.816259 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:33.816231    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf"]
Sep 15 11:39:33.868125 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:33.868090    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c2b4fa9-7e08-446a-a03b-1613a8ab4f98-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf\" (UID: \"6c2b4fa9-7e08-446a-a03b-1613a8ab4f98\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf"
Sep 15 11:39:33.868325 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:33.868194    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzsnh\" (UniqueName: \"kubernetes.io/projected/6c2b4fa9-7e08-446a-a03b-1613a8ab4f98-kube-api-access-xzsnh\") pod \"payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf\" (UID: \"6c2b4fa9-7e08-446a-a03b-1613a8ab4f98\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf"
Sep 15 11:39:33.969357 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:33.969320    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c2b4fa9-7e08-446a-a03b-1613a8ab4f98-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf\" (UID: \"6c2b4fa9-7e08-446a-a03b-1613a8ab4f98\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf"
Sep 15 11:39:33.969547 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:33.969409    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzsnh\" (UniqueName: \"kubernetes.io/projected/6c2b4fa9-7e08-446a-a03b-1613a8ab4f98-kube-api-access-xzsnh\") pod \"payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf\" (UID: \"6c2b4fa9-7e08-446a-a03b-1613a8ab4f98\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf"
Sep 15 11:39:33.970007 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:33.969982    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c2b4fa9-7e08-446a-a03b-1613a8ab4f98-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf\" (UID: \"6c2b4fa9-7e08-446a-a03b-1613a8ab4f98\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf"
Sep 15 11:39:33.985599 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:33.985565    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzsnh\" (UniqueName: \"kubernetes.io/projected/6c2b4fa9-7e08-446a-a03b-1613a8ab4f98-kube-api-access-xzsnh\") pod \"payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf\" (UID: \"6c2b4fa9-7e08-446a-a03b-1613a8ab4f98\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf"
Sep 15 11:39:34.092760 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:34.092724    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf"
Sep 15 11:39:34.362181 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:34.361977    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf"]
Sep 15 11:39:34.365231 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:39:34.365195    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2b4fa9_7e08_446a_a03b_1613a8ab4f98.slice/crio-c11a50f99f92031c16c6f904165e08f591c99267ec243944f6693d69ea8994ff WatchSource:0}: Error finding container c11a50f99f92031c16c6f904165e08f591c99267ec243944f6693d69ea8994ff: Status 404 returned error can't find the container with id c11a50f99f92031c16c6f904165e08f591c99267ec243944f6693d69ea8994ff
Sep 15 11:39:34.592134 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:34.592098    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:39:34.593266 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:34.593236    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf" event={"ID":"6c2b4fa9-7e08-446a-a03b-1613a8ab4f98","Type":"ContainerStarted","Data":"356e339da5c76a625fab34ced8a056b9a7a4ce0f43b028c519a934a03d4f80ba"}
Sep 15 11:39:34.593576 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:34.593277    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf" event={"ID":"6c2b4fa9-7e08-446a-a03b-1613a8ab4f98","Type":"ContainerStarted","Data":"c11a50f99f92031c16c6f904165e08f591c99267ec243944f6693d69ea8994ff"}
Sep 15 11:39:34.593576 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:34.593360    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf"
Sep 15 11:39:34.647606 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:34.647485    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf" podStartSLOduration=1.6474640219999999 podStartE2EDuration="1.647464022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:34.646791782 +0000 UTC m=+873.309286983" watchObservedRunningTime="2026-09-15 11:39:34.647464022 +0000 UTC m=+873.309959204"
Sep 15 11:39:45.600859 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:45.600824    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf"
Sep 15 11:39:55.560860 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:39:55.560764    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:40:12.628202 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:12.628165    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"]
Sep 15 11:40:12.628701 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:12.628417    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7" podUID="62710c30-a12e-467b-bdca-d4490c646683" containerName="maas-api" containerID="cri-o://4f0249506408b20a6f11921b707fa66a780769d9438ba9e585811526c1a1dd64" gracePeriod=30
Sep 15 11:40:12.695142 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:12.695099    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl"]
Sep 15 11:40:12.695436 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:12.695405    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl" podUID="f8ff4115-8602-42ab-804a-3098defbc45c" containerName="payload-processing" containerID="cri-o://b240bb6c7683fe51bd6b01506330b7cc4fc66b7dfb716883cb1310acb8b59352" gracePeriod=30
Sep 15 11:40:12.781319 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:12.781286    2562 generic.go:356] "Generic (PLEG): container finished" podID="62710c30-a12e-467b-bdca-d4490c646683" containerID="4f0249506408b20a6f11921b707fa66a780769d9438ba9e585811526c1a1dd64" exitCode=0
Sep 15 11:40:12.781515 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:12.781362    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7" event={"ID":"62710c30-a12e-467b-bdca-d4490c646683","Type":"ContainerDied","Data":"4f0249506408b20a6f11921b707fa66a780769d9438ba9e585811526c1a1dd64"}
Sep 15 11:40:12.897238 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:12.897154    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf"]
Sep 15 11:40:12.897448 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:12.897416    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf" podUID="6c2b4fa9-7e08-446a-a03b-1613a8ab4f98" containerName="payload-pre-processing" containerID="cri-o://356e339da5c76a625fab34ced8a056b9a7a4ce0f43b028c519a934a03d4f80ba" gracePeriod=30
Sep 15 11:40:13.011601 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.011563    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"
Sep 15 11:40:13.013863 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.013839    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl"
Sep 15 11:40:13.123589 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.123554    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rx6xd\" (UniqueName: \"kubernetes.io/projected/f8ff4115-8602-42ab-804a-3098defbc45c-kube-api-access-rx6xd\") pod \"f8ff4115-8602-42ab-804a-3098defbc45c\" (UID: \"f8ff4115-8602-42ab-804a-3098defbc45c\") "
Sep 15 11:40:13.123775 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.123610    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zv46\" (UniqueName: \"kubernetes.io/projected/62710c30-a12e-467b-bdca-d4490c646683-kube-api-access-2zv46\") pod \"62710c30-a12e-467b-bdca-d4490c646683\" (UID: \"62710c30-a12e-467b-bdca-d4490c646683\") "
Sep 15 11:40:13.123775 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.123637    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62710c30-a12e-467b-bdca-d4490c646683-openshift-service-ca\") pod \"62710c30-a12e-467b-bdca-d4490c646683\" (UID: \"62710c30-a12e-467b-bdca-d4490c646683\") "
Sep 15 11:40:13.123775 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.123655    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62710c30-a12e-467b-bdca-d4490c646683-maas-api-tls\") pod \"62710c30-a12e-467b-bdca-d4490c646683\" (UID: \"62710c30-a12e-467b-bdca-d4490c646683\") "
Sep 15 11:40:13.123775 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.123696    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ff4115-8602-42ab-804a-3098defbc45c-plugins-config-volume\") pod \"f8ff4115-8602-42ab-804a-3098defbc45c\" (UID: \"f8ff4115-8602-42ab-804a-3098defbc45c\") "
Sep 15 11:40:13.124323 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.124004    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62710c30-a12e-467b-bdca-d4490c646683-metrics-certs\") pod \"62710c30-a12e-467b-bdca-d4490c646683\" (UID: \"62710c30-a12e-467b-bdca-d4490c646683\") "
Sep 15 11:40:13.124323 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.124088    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62710c30-a12e-467b-bdca-d4490c646683-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "62710c30-a12e-467b-bdca-d4490c646683" (UID: "62710c30-a12e-467b-bdca-d4490c646683"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:40:13.124323 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.124120    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ff4115-8602-42ab-804a-3098defbc45c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8ff4115-8602-42ab-804a-3098defbc45c" (UID: "f8ff4115-8602-42ab-804a-3098defbc45c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:40:13.124542 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.124330    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62710c30-a12e-467b-bdca-d4490c646683-openshift-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:40:13.124542 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.124352    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8ff4115-8602-42ab-804a-3098defbc45c-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:40:13.126738 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.126710    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62710c30-a12e-467b-bdca-d4490c646683-kube-api-access-2zv46" (OuterVolumeSpecName: "kube-api-access-2zv46") pod "62710c30-a12e-467b-bdca-d4490c646683" (UID: "62710c30-a12e-467b-bdca-d4490c646683"). InnerVolumeSpecName "kube-api-access-2zv46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:40:13.126738 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.126725    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62710c30-a12e-467b-bdca-d4490c646683-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "62710c30-a12e-467b-bdca-d4490c646683" (UID: "62710c30-a12e-467b-bdca-d4490c646683"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:40:13.126862 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.126708    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62710c30-a12e-467b-bdca-d4490c646683-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "62710c30-a12e-467b-bdca-d4490c646683" (UID: "62710c30-a12e-467b-bdca-d4490c646683"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:40:13.126862 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.126807    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ff4115-8602-42ab-804a-3098defbc45c-kube-api-access-rx6xd" (OuterVolumeSpecName: "kube-api-access-rx6xd") pod "f8ff4115-8602-42ab-804a-3098defbc45c" (UID: "f8ff4115-8602-42ab-804a-3098defbc45c"). InnerVolumeSpecName "kube-api-access-rx6xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:40:13.155975 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.155920    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf"
Sep 15 11:40:13.225526 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.225497    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c2b4fa9-7e08-446a-a03b-1613a8ab4f98-plugins-config-volume\") pod \"6c2b4fa9-7e08-446a-a03b-1613a8ab4f98\" (UID: \"6c2b4fa9-7e08-446a-a03b-1613a8ab4f98\") "
Sep 15 11:40:13.225717 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.225546    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzsnh\" (UniqueName: \"kubernetes.io/projected/6c2b4fa9-7e08-446a-a03b-1613a8ab4f98-kube-api-access-xzsnh\") pod \"6c2b4fa9-7e08-446a-a03b-1613a8ab4f98\" (UID: \"6c2b4fa9-7e08-446a-a03b-1613a8ab4f98\") "
Sep 15 11:40:13.225788 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.225754    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62710c30-a12e-467b-bdca-d4490c646683-metrics-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:40:13.225788 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.225766    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx6xd\" (UniqueName: \"kubernetes.io/projected/f8ff4115-8602-42ab-804a-3098defbc45c-kube-api-access-rx6xd\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:40:13.225788 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.225777    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zv46\" (UniqueName: \"kubernetes.io/projected/62710c30-a12e-467b-bdca-d4490c646683-kube-api-access-2zv46\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:40:13.225942 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.225791    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62710c30-a12e-467b-bdca-d4490c646683-maas-api-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:40:13.225942 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.225856    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c2b4fa9-7e08-446a-a03b-1613a8ab4f98-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c2b4fa9-7e08-446a-a03b-1613a8ab4f98" (UID: "6c2b4fa9-7e08-446a-a03b-1613a8ab4f98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:40:13.227745 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.227715    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2b4fa9-7e08-446a-a03b-1613a8ab4f98-kube-api-access-xzsnh" (OuterVolumeSpecName: "kube-api-access-xzsnh") pod "6c2b4fa9-7e08-446a-a03b-1613a8ab4f98" (UID: "6c2b4fa9-7e08-446a-a03b-1613a8ab4f98"). InnerVolumeSpecName "kube-api-access-xzsnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:40:13.326424 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.326387    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzsnh\" (UniqueName: \"kubernetes.io/projected/6c2b4fa9-7e08-446a-a03b-1613a8ab4f98-kube-api-access-xzsnh\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:40:13.326424 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.326416    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c2b4fa9-7e08-446a-a03b-1613a8ab4f98-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:40:13.787859 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.787827    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"
Sep 15 11:40:13.788381 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.787830    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7" event={"ID":"62710c30-a12e-467b-bdca-d4490c646683","Type":"ContainerDied","Data":"626aefa1e3e2158f51d3853890b4d503a73661f755c58788034197c8e451a974"}
Sep 15 11:40:13.788381 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.787975    2562 scope.go:117] "RemoveContainer" containerID="4f0249506408b20a6f11921b707fa66a780769d9438ba9e585811526c1a1dd64"
Sep 15 11:40:13.789493 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.789449    2562 generic.go:356] "Generic (PLEG): container finished" podID="6c2b4fa9-7e08-446a-a03b-1613a8ab4f98" containerID="356e339da5c76a625fab34ced8a056b9a7a4ce0f43b028c519a934a03d4f80ba" exitCode=0
Sep 15 11:40:13.789590 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.789499    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf"
Sep 15 11:40:13.789590 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.789500    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf" event={"ID":"6c2b4fa9-7e08-446a-a03b-1613a8ab4f98","Type":"ContainerDied","Data":"356e339da5c76a625fab34ced8a056b9a7a4ce0f43b028c519a934a03d4f80ba"}
Sep 15 11:40:13.789590 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.789553    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf" event={"ID":"6c2b4fa9-7e08-446a-a03b-1613a8ab4f98","Type":"ContainerDied","Data":"c11a50f99f92031c16c6f904165e08f591c99267ec243944f6693d69ea8994ff"}
Sep 15 11:40:13.791030 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.791003    2562 generic.go:356] "Generic (PLEG): container finished" podID="f8ff4115-8602-42ab-804a-3098defbc45c" containerID="b240bb6c7683fe51bd6b01506330b7cc4fc66b7dfb716883cb1310acb8b59352" exitCode=0
Sep 15 11:40:13.791172 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.791034    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl"
Sep 15 11:40:13.791172 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.791081    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl" event={"ID":"f8ff4115-8602-42ab-804a-3098defbc45c","Type":"ContainerDied","Data":"b240bb6c7683fe51bd6b01506330b7cc4fc66b7dfb716883cb1310acb8b59352"}
Sep 15 11:40:13.791172 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.791128    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl" event={"ID":"f8ff4115-8602-42ab-804a-3098defbc45c","Type":"ContainerDied","Data":"b041662a8623fb9f9763824c79b4690ee2c62f292440570909014336bddd88bd"}
Sep 15 11:40:13.798290 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.798276    2562 scope.go:117] "RemoveContainer" containerID="356e339da5c76a625fab34ced8a056b9a7a4ce0f43b028c519a934a03d4f80ba"
Sep 15 11:40:13.806927 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.806905    2562 scope.go:117] "RemoveContainer" containerID="356e339da5c76a625fab34ced8a056b9a7a4ce0f43b028c519a934a03d4f80ba"
Sep 15 11:40:13.807237 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:40:13.807220    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"356e339da5c76a625fab34ced8a056b9a7a4ce0f43b028c519a934a03d4f80ba\": container with ID starting with 356e339da5c76a625fab34ced8a056b9a7a4ce0f43b028c519a934a03d4f80ba not found: ID does not exist" containerID="356e339da5c76a625fab34ced8a056b9a7a4ce0f43b028c519a934a03d4f80ba"
Sep 15 11:40:13.807317 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.807243    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"356e339da5c76a625fab34ced8a056b9a7a4ce0f43b028c519a934a03d4f80ba"} err="failed to get container status \"356e339da5c76a625fab34ced8a056b9a7a4ce0f43b028c519a934a03d4f80ba\": rpc error: code = NotFound desc = could not find container \"356e339da5c76a625fab34ced8a056b9a7a4ce0f43b028c519a934a03d4f80ba\": container with ID starting with 356e339da5c76a625fab34ced8a056b9a7a4ce0f43b028c519a934a03d4f80ba not found: ID does not exist"
Sep 15 11:40:13.807317 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.807260    2562 scope.go:117] "RemoveContainer" containerID="b240bb6c7683fe51bd6b01506330b7cc4fc66b7dfb716883cb1310acb8b59352"
Sep 15 11:40:13.813601 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.813573    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"]
Sep 15 11:40:13.815514 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.815492    2562 scope.go:117] "RemoveContainer" containerID="b240bb6c7683fe51bd6b01506330b7cc4fc66b7dfb716883cb1310acb8b59352"
Sep 15 11:40:13.816039 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:40:13.815865    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b240bb6c7683fe51bd6b01506330b7cc4fc66b7dfb716883cb1310acb8b59352\": container with ID starting with b240bb6c7683fe51bd6b01506330b7cc4fc66b7dfb716883cb1310acb8b59352 not found: ID does not exist" containerID="b240bb6c7683fe51bd6b01506330b7cc4fc66b7dfb716883cb1310acb8b59352"
Sep 15 11:40:13.816039 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.815901    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b240bb6c7683fe51bd6b01506330b7cc4fc66b7dfb716883cb1310acb8b59352"} err="failed to get container status \"b240bb6c7683fe51bd6b01506330b7cc4fc66b7dfb716883cb1310acb8b59352\": rpc error: code = NotFound desc = could not find container \"b240bb6c7683fe51bd6b01506330b7cc4fc66b7dfb716883cb1310acb8b59352\": container with ID starting with b240bb6c7683fe51bd6b01506330b7cc4fc66b7dfb716883cb1310acb8b59352 not found: ID does not exist"
Sep 15 11:40:13.817974 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.817952    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4d6dbb-7f6dbbb84d-449b7"]
Sep 15 11:40:13.828496 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.828474    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf"]
Sep 15 11:40:13.835079 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.835046    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9b1cbda7-6f9dd44778-6rhnf"]
Sep 15 11:40:13.847362 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.847342    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl"]
Sep 15 11:40:13.851732 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.851705    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4d6dbb-77fc667d85-ppldl"]
Sep 15 11:40:13.879465 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.875230    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62710c30-a12e-467b-bdca-d4490c646683" path="/var/lib/kubelet/pods/62710c30-a12e-467b-bdca-d4490c646683/volumes"
Sep 15 11:40:13.879465 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.875810    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c2b4fa9-7e08-446a-a03b-1613a8ab4f98" path="/var/lib/kubelet/pods/6c2b4fa9-7e08-446a-a03b-1613a8ab4f98/volumes"
Sep 15 11:40:13.879465 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:13.876323    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8ff4115-8602-42ab-804a-3098defbc45c" path="/var/lib/kubelet/pods/f8ff4115-8602-42ab-804a-3098defbc45c/volumes"
Sep 15 11:40:20.961263 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:20.961220    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:40:22.021146 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.021106    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"]
Sep 15 11:40:22.021525 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.021508    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c2b4fa9-7e08-446a-a03b-1613a8ab4f98" containerName="payload-pre-processing"
Sep 15 11:40:22.021525 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.021523    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c2b4fa9-7e08-446a-a03b-1613a8ab4f98" containerName="payload-pre-processing"
Sep 15 11:40:22.021605 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.021538    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62710c30-a12e-467b-bdca-d4490c646683" containerName="maas-api"
Sep 15 11:40:22.021605 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.021544    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="62710c30-a12e-467b-bdca-d4490c646683" containerName="maas-api"
Sep 15 11:40:22.021605 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.021553    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8ff4115-8602-42ab-804a-3098defbc45c" containerName="payload-processing"
Sep 15 11:40:22.021605 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.021560    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8ff4115-8602-42ab-804a-3098defbc45c" containerName="payload-processing"
Sep 15 11:40:22.021726 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.021622    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="62710c30-a12e-467b-bdca-d4490c646683" containerName="maas-api"
Sep 15 11:40:22.021726 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.021630    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8ff4115-8602-42ab-804a-3098defbc45c" containerName="payload-processing"
Sep 15 11:40:22.021726 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.021636    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c2b4fa9-7e08-446a-a03b-1613a8ab4f98" containerName="payload-pre-processing"
Sep 15 11:40:22.026435 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.026417    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"
Sep 15 11:40:22.029222 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.029195    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-6f21996a-serving-cert\""
Sep 15 11:40:22.038369 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.038334    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"]
Sep 15 11:40:22.104738 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.104700    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6hbk\" (UniqueName: \"kubernetes.io/projected/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-kube-api-access-s6hbk\") pod \"maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp\" (UID: \"8383a7d3-8b1e-429a-8dae-2dbb150a00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"
Sep 15 11:40:22.104944 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.104773    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-openshift-service-ca\") pod \"maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp\" (UID: \"8383a7d3-8b1e-429a-8dae-2dbb150a00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"
Sep 15 11:40:22.104944 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.104838    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-maas-api-tls\") pod \"maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp\" (UID: \"8383a7d3-8b1e-429a-8dae-2dbb150a00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"
Sep 15 11:40:22.104944 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.104876    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-metrics-certs\") pod \"maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp\" (UID: \"8383a7d3-8b1e-429a-8dae-2dbb150a00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"
Sep 15 11:40:22.205501 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.205465    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-metrics-certs\") pod \"maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp\" (UID: \"8383a7d3-8b1e-429a-8dae-2dbb150a00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"
Sep 15 11:40:22.205694 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.205510    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6hbk\" (UniqueName: \"kubernetes.io/projected/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-kube-api-access-s6hbk\") pod \"maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp\" (UID: \"8383a7d3-8b1e-429a-8dae-2dbb150a00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"
Sep 15 11:40:22.205694 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.205557    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-openshift-service-ca\") pod \"maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp\" (UID: \"8383a7d3-8b1e-429a-8dae-2dbb150a00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"
Sep 15 11:40:22.205694 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.205579    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-maas-api-tls\") pod \"maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp\" (UID: \"8383a7d3-8b1e-429a-8dae-2dbb150a00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"
Sep 15 11:40:22.206294 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.206267    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-openshift-service-ca\") pod \"maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp\" (UID: \"8383a7d3-8b1e-429a-8dae-2dbb150a00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"
Sep 15 11:40:22.208012 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.207979    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-maas-api-tls\") pod \"maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp\" (UID: \"8383a7d3-8b1e-429a-8dae-2dbb150a00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"
Sep 15 11:40:22.208110 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.208071    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-metrics-certs\") pod \"maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp\" (UID: \"8383a7d3-8b1e-429a-8dae-2dbb150a00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"
Sep 15 11:40:22.213256 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.213236    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6hbk\" (UniqueName: \"kubernetes.io/projected/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-kube-api-access-s6hbk\") pod \"maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp\" (UID: \"8383a7d3-8b1e-429a-8dae-2dbb150a00fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"
Sep 15 11:40:22.338118 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.338081    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"
Sep 15 11:40:22.617293 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.617216    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4"]
Sep 15 11:40:22.621619 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:40:22.621591    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8383a7d3_8b1e_429a_8dae_2dbb150a00fc.slice/crio-a33000c949e8745932bd26aba75b2c4f633a4a87a57aee73c2f4d16e8fcb1291 WatchSource:0}: Error finding container a33000c949e8745932bd26aba75b2c4f633a4a87a57aee73c2f4d16e8fcb1291: Status 404 returned error can't find the container with id a33000c949e8745932bd26aba75b2c4f633a4a87a57aee73c2f4d16e8fcb1291
Sep 15 11:40:22.621831 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.621814    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"]
Sep 15 11:40:22.621911 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.621901    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4"
Sep 15 11:40:22.624392 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.624369    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-6f21996a\""
Sep 15 11:40:22.624666 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.624639    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-6f21996a-dockercfg-nw4gw\""
Sep 15 11:40:22.634916 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.634832    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4"]
Sep 15 11:40:22.666219 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.666186    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx"]
Sep 15 11:40:22.669467 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.669451    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx"
Sep 15 11:40:22.682546 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.682520    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx"]
Sep 15 11:40:22.710440 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.710408    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26fb3833-f208-48c9-8f0b-1ec2c6ed81d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4\" (UID: \"26fb3833-f208-48c9-8f0b-1ec2c6ed81d8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4"
Sep 15 11:40:22.710568 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.710467    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfkkv\" (UniqueName: \"kubernetes.io/projected/26fb3833-f208-48c9-8f0b-1ec2c6ed81d8-kube-api-access-nfkkv\") pod \"payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4\" (UID: \"26fb3833-f208-48c9-8f0b-1ec2c6ed81d8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4"
Sep 15 11:40:22.811116 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.811029    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26fb3833-f208-48c9-8f0b-1ec2c6ed81d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4\" (UID: \"26fb3833-f208-48c9-8f0b-1ec2c6ed81d8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4"
Sep 15 11:40:22.811314 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.811132    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvmzn\" (UniqueName: \"kubernetes.io/projected/18c700e8-a0fd-44ae-9232-69a88e161e90-kube-api-access-lvmzn\") pod \"payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx\" (UID: \"18c700e8-a0fd-44ae-9232-69a88e161e90\") " pod="openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx"
Sep 15 11:40:22.811314 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.811178    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c700e8-a0fd-44ae-9232-69a88e161e90-plugins-config-volume\") pod \"payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx\" (UID: \"18c700e8-a0fd-44ae-9232-69a88e161e90\") " pod="openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx"
Sep 15 11:40:22.811314 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.811200    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfkkv\" (UniqueName: \"kubernetes.io/projected/26fb3833-f208-48c9-8f0b-1ec2c6ed81d8-kube-api-access-nfkkv\") pod \"payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4\" (UID: \"26fb3833-f208-48c9-8f0b-1ec2c6ed81d8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4"
Sep 15 11:40:22.811747 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.811726    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26fb3833-f208-48c9-8f0b-1ec2c6ed81d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4\" (UID: \"26fb3833-f208-48c9-8f0b-1ec2c6ed81d8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4"
Sep 15 11:40:22.819473 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.819439    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfkkv\" (UniqueName: \"kubernetes.io/projected/26fb3833-f208-48c9-8f0b-1ec2c6ed81d8-kube-api-access-nfkkv\") pod \"payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4\" (UID: \"26fb3833-f208-48c9-8f0b-1ec2c6ed81d8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4"
Sep 15 11:40:22.835923 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.835885    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp" event={"ID":"8383a7d3-8b1e-429a-8dae-2dbb150a00fc","Type":"ContainerStarted","Data":"a33000c949e8745932bd26aba75b2c4f633a4a87a57aee73c2f4d16e8fcb1291"}
Sep 15 11:40:22.912598 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.912493    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvmzn\" (UniqueName: \"kubernetes.io/projected/18c700e8-a0fd-44ae-9232-69a88e161e90-kube-api-access-lvmzn\") pod \"payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx\" (UID: \"18c700e8-a0fd-44ae-9232-69a88e161e90\") " pod="openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx"
Sep 15 11:40:22.912598 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.912583    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c700e8-a0fd-44ae-9232-69a88e161e90-plugins-config-volume\") pod \"payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx\" (UID: \"18c700e8-a0fd-44ae-9232-69a88e161e90\") " pod="openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx"
Sep 15 11:40:22.913348 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.913329    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c700e8-a0fd-44ae-9232-69a88e161e90-plugins-config-volume\") pod \"payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx\" (UID: \"18c700e8-a0fd-44ae-9232-69a88e161e90\") " pod="openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx"
Sep 15 11:40:22.920528 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.920509    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvmzn\" (UniqueName: \"kubernetes.io/projected/18c700e8-a0fd-44ae-9232-69a88e161e90-kube-api-access-lvmzn\") pod \"payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx\" (UID: \"18c700e8-a0fd-44ae-9232-69a88e161e90\") " pod="openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx"
Sep 15 11:40:22.940483 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.940456    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4"
Sep 15 11:40:22.979655 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:22.979619    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx"
Sep 15 11:40:23.212040 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:23.212016    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4"]
Sep 15 11:40:23.213680 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:40:23.213653    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fb3833_f208_48c9_8f0b_1ec2c6ed81d8.slice/crio-7b325fbd2ee61bb0edb7fb06a653e3140ba701528e2fd76f21f9e45baf4474ef WatchSource:0}: Error finding container 7b325fbd2ee61bb0edb7fb06a653e3140ba701528e2fd76f21f9e45baf4474ef: Status 404 returned error can't find the container with id 7b325fbd2ee61bb0edb7fb06a653e3140ba701528e2fd76f21f9e45baf4474ef
Sep 15 11:40:23.264876 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:23.264846    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx"]
Sep 15 11:40:23.268714 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:40:23.268648    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c700e8_a0fd_44ae_9232_69a88e161e90.slice/crio-4d177e48249a754da22d8f135f673d1328d87575f7e17370e74a0cfece01dc47 WatchSource:0}: Error finding container 4d177e48249a754da22d8f135f673d1328d87575f7e17370e74a0cfece01dc47: Status 404 returned error can't find the container with id 4d177e48249a754da22d8f135f673d1328d87575f7e17370e74a0cfece01dc47
Sep 15 11:40:23.843706 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:23.843664    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx" event={"ID":"18c700e8-a0fd-44ae-9232-69a88e161e90","Type":"ContainerStarted","Data":"e83e9d951d6f157e847035b5c15c25af1e6b1ead319cb443a8849444c4b2d3cf"}
Sep 15 11:40:23.843900 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:23.843760    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx" event={"ID":"18c700e8-a0fd-44ae-9232-69a88e161e90","Type":"ContainerStarted","Data":"4d177e48249a754da22d8f135f673d1328d87575f7e17370e74a0cfece01dc47"}
Sep 15 11:40:23.843900 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:23.843801    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx"
Sep 15 11:40:23.845434 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:23.845403    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4" event={"ID":"26fb3833-f208-48c9-8f0b-1ec2c6ed81d8","Type":"ContainerStarted","Data":"2cae0125cfe3c1dc34176baa6f1adcb8184b70a2b048a67a256f81daee818afc"}
Sep 15 11:40:23.845547 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:23.845463    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4"
Sep 15 11:40:23.845547 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:23.845482    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4" event={"ID":"26fb3833-f208-48c9-8f0b-1ec2c6ed81d8","Type":"ContainerStarted","Data":"7b325fbd2ee61bb0edb7fb06a653e3140ba701528e2fd76f21f9e45baf4474ef"}
Sep 15 11:40:23.847021 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:23.846997    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp" event={"ID":"8383a7d3-8b1e-429a-8dae-2dbb150a00fc","Type":"ContainerStarted","Data":"28e97e226f04e94e297303549528fa06b5abdec5c84334e0e0669247323082d8"}
Sep 15 11:40:23.847189 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:23.847174    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"
Sep 15 11:40:23.863910 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:23.863856    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx" podStartSLOduration=1.863837832 podStartE2EDuration="1.863837832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:23.861811812 +0000 UTC m=+922.524307016" watchObservedRunningTime="2026-09-15 11:40:23.863837832 +0000 UTC m=+922.526333014"
Sep 15 11:40:23.882411 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:23.882222    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp" podStartSLOduration=2.300139622 podStartE2EDuration="2.88220403s" totalImagesPullingTime="582.064408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:23.878456478 +0000 UTC m=+922.540951681" watchObservedRunningTime="2026-09-15 11:40:23.88220403 +0000 UTC m=+922.544699215"
Sep 15 11:40:23.896558 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:23.896492    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4" podStartSLOduration=1.896478814 podStartE2EDuration="1.896478814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:23.895582684 +0000 UTC m=+922.558077886" watchObservedRunningTime="2026-09-15 11:40:23.896478814 +0000 UTC m=+922.558973995"
Sep 15 11:40:29.863359 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:29.863324    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"
Sep 15 11:40:34.853604 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:34.853573    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx"
Sep 15 11:40:34.856042 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:34.853647    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4"
Sep 15 11:40:47.591227 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:40:47.591191    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:41:00.031975 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.031938    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"]
Sep 15 11:41:00.032448 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.032156    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp" podUID="8383a7d3-8b1e-429a-8dae-2dbb150a00fc" containerName="maas-api" containerID="cri-o://28e97e226f04e94e297303549528fa06b5abdec5c84334e0e0669247323082d8" gracePeriod=30
Sep 15 11:41:00.115277 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.115240    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx"]
Sep 15 11:41:00.115490 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.115464    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx" podUID="18c700e8-a0fd-44ae-9232-69a88e161e90" containerName="payload-processing" containerID="cri-o://e83e9d951d6f157e847035b5c15c25af1e6b1ead319cb443a8849444c4b2d3cf" gracePeriod=30
Sep 15 11:41:00.127012 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.126902    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4"]
Sep 15 11:41:00.127476 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.127403    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4" podUID="26fb3833-f208-48c9-8f0b-1ec2c6ed81d8" containerName="payload-pre-processing" containerID="cri-o://2cae0125cfe3c1dc34176baa6f1adcb8184b70a2b048a67a256f81daee818afc" gracePeriod=30
Sep 15 11:41:00.376037 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.376007    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"
Sep 15 11:41:00.414571 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.414546    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx"
Sep 15 11:41:00.432748 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.432524    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4"
Sep 15 11:41:00.447776 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.447750    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6hbk\" (UniqueName: \"kubernetes.io/projected/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-kube-api-access-s6hbk\") pod \"8383a7d3-8b1e-429a-8dae-2dbb150a00fc\" (UID: \"8383a7d3-8b1e-429a-8dae-2dbb150a00fc\") "
Sep 15 11:41:00.447926 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.447807    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-openshift-service-ca\") pod \"8383a7d3-8b1e-429a-8dae-2dbb150a00fc\" (UID: \"8383a7d3-8b1e-429a-8dae-2dbb150a00fc\") "
Sep 15 11:41:00.447926 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.447874    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-maas-api-tls\") pod \"8383a7d3-8b1e-429a-8dae-2dbb150a00fc\" (UID: \"8383a7d3-8b1e-429a-8dae-2dbb150a00fc\") "
Sep 15 11:41:00.447926 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.447909    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-metrics-certs\") pod \"8383a7d3-8b1e-429a-8dae-2dbb150a00fc\" (UID: \"8383a7d3-8b1e-429a-8dae-2dbb150a00fc\") "
Sep 15 11:41:00.448208 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.448185    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8383a7d3-8b1e-429a-8dae-2dbb150a00fc" (UID: "8383a7d3-8b1e-429a-8dae-2dbb150a00fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:41:00.450047 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.450022    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8383a7d3-8b1e-429a-8dae-2dbb150a00fc" (UID: "8383a7d3-8b1e-429a-8dae-2dbb150a00fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:41:00.450047 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.450051    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8383a7d3-8b1e-429a-8dae-2dbb150a00fc" (UID: "8383a7d3-8b1e-429a-8dae-2dbb150a00fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:41:00.450195 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.450070    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-kube-api-access-s6hbk" (OuterVolumeSpecName: "kube-api-access-s6hbk") pod "8383a7d3-8b1e-429a-8dae-2dbb150a00fc" (UID: "8383a7d3-8b1e-429a-8dae-2dbb150a00fc"). InnerVolumeSpecName "kube-api-access-s6hbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:41:00.549291 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.549253    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c700e8-a0fd-44ae-9232-69a88e161e90-plugins-config-volume\") pod \"18c700e8-a0fd-44ae-9232-69a88e161e90\" (UID: \"18c700e8-a0fd-44ae-9232-69a88e161e90\") "
Sep 15 11:41:00.549291 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.549295    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfkkv\" (UniqueName: \"kubernetes.io/projected/26fb3833-f208-48c9-8f0b-1ec2c6ed81d8-kube-api-access-nfkkv\") pod \"26fb3833-f208-48c9-8f0b-1ec2c6ed81d8\" (UID: \"26fb3833-f208-48c9-8f0b-1ec2c6ed81d8\") "
Sep 15 11:41:00.549552 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.549329    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvmzn\" (UniqueName: \"kubernetes.io/projected/18c700e8-a0fd-44ae-9232-69a88e161e90-kube-api-access-lvmzn\") pod \"18c700e8-a0fd-44ae-9232-69a88e161e90\" (UID: \"18c700e8-a0fd-44ae-9232-69a88e161e90\") "
Sep 15 11:41:00.549552 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.549368    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26fb3833-f208-48c9-8f0b-1ec2c6ed81d8-plugins-config-volume\") pod \"26fb3833-f208-48c9-8f0b-1ec2c6ed81d8\" (UID: \"26fb3833-f208-48c9-8f0b-1ec2c6ed81d8\") "
Sep 15 11:41:00.549658 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.549572    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-openshift-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:41:00.549658 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.549593    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-maas-api-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:41:00.549658 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.549607    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-metrics-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:41:00.549658 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.549620    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6hbk\" (UniqueName: \"kubernetes.io/projected/8383a7d3-8b1e-429a-8dae-2dbb150a00fc-kube-api-access-s6hbk\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:41:00.549823 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.549791    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c700e8-a0fd-44ae-9232-69a88e161e90-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "18c700e8-a0fd-44ae-9232-69a88e161e90" (UID: "18c700e8-a0fd-44ae-9232-69a88e161e90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:41:00.549865 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.549823    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26fb3833-f208-48c9-8f0b-1ec2c6ed81d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "26fb3833-f208-48c9-8f0b-1ec2c6ed81d8" (UID: "26fb3833-f208-48c9-8f0b-1ec2c6ed81d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:41:00.551490 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.551466    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fb3833-f208-48c9-8f0b-1ec2c6ed81d8-kube-api-access-nfkkv" (OuterVolumeSpecName: "kube-api-access-nfkkv") pod "26fb3833-f208-48c9-8f0b-1ec2c6ed81d8" (UID: "26fb3833-f208-48c9-8f0b-1ec2c6ed81d8"). InnerVolumeSpecName "kube-api-access-nfkkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:41:00.551559 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.551512    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c700e8-a0fd-44ae-9232-69a88e161e90-kube-api-access-lvmzn" (OuterVolumeSpecName: "kube-api-access-lvmzn") pod "18c700e8-a0fd-44ae-9232-69a88e161e90" (UID: "18c700e8-a0fd-44ae-9232-69a88e161e90"). InnerVolumeSpecName "kube-api-access-lvmzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:41:00.650965 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.650882    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c700e8-a0fd-44ae-9232-69a88e161e90-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:41:00.650965 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.650908    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfkkv\" (UniqueName: \"kubernetes.io/projected/26fb3833-f208-48c9-8f0b-1ec2c6ed81d8-kube-api-access-nfkkv\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:41:00.650965 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.650918    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvmzn\" (UniqueName: \"kubernetes.io/projected/18c700e8-a0fd-44ae-9232-69a88e161e90-kube-api-access-lvmzn\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:41:00.650965 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:00.650926    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26fb3833-f208-48c9-8f0b-1ec2c6ed81d8-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:41:01.020594 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.020514    2562 generic.go:356] "Generic (PLEG): container finished" podID="18c700e8-a0fd-44ae-9232-69a88e161e90" containerID="e83e9d951d6f157e847035b5c15c25af1e6b1ead319cb443a8849444c4b2d3cf" exitCode=0
Sep 15 11:41:01.020594 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.020558    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx"
Sep 15 11:41:01.020594 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.020583    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx" event={"ID":"18c700e8-a0fd-44ae-9232-69a88e161e90","Type":"ContainerDied","Data":"e83e9d951d6f157e847035b5c15c25af1e6b1ead319cb443a8849444c4b2d3cf"}
Sep 15 11:41:01.020835 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.020638    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx" event={"ID":"18c700e8-a0fd-44ae-9232-69a88e161e90","Type":"ContainerDied","Data":"4d177e48249a754da22d8f135f673d1328d87575f7e17370e74a0cfece01dc47"}
Sep 15 11:41:01.020835 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.020670    2562 scope.go:117] "RemoveContainer" containerID="e83e9d951d6f157e847035b5c15c25af1e6b1ead319cb443a8849444c4b2d3cf"
Sep 15 11:41:01.022072 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.022020    2562 generic.go:356] "Generic (PLEG): container finished" podID="26fb3833-f208-48c9-8f0b-1ec2c6ed81d8" containerID="2cae0125cfe3c1dc34176baa6f1adcb8184b70a2b048a67a256f81daee818afc" exitCode=0
Sep 15 11:41:01.022152 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.022087    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4"
Sep 15 11:41:01.022152 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.022085    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4" event={"ID":"26fb3833-f208-48c9-8f0b-1ec2c6ed81d8","Type":"ContainerDied","Data":"2cae0125cfe3c1dc34176baa6f1adcb8184b70a2b048a67a256f81daee818afc"}
Sep 15 11:41:01.022267 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.022191    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4" event={"ID":"26fb3833-f208-48c9-8f0b-1ec2c6ed81d8","Type":"ContainerDied","Data":"7b325fbd2ee61bb0edb7fb06a653e3140ba701528e2fd76f21f9e45baf4474ef"}
Sep 15 11:41:01.023763 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.023740    2562 generic.go:356] "Generic (PLEG): container finished" podID="8383a7d3-8b1e-429a-8dae-2dbb150a00fc" containerID="28e97e226f04e94e297303549528fa06b5abdec5c84334e0e0669247323082d8" exitCode=0
Sep 15 11:41:01.023943 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.023765    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp" event={"ID":"8383a7d3-8b1e-429a-8dae-2dbb150a00fc","Type":"ContainerDied","Data":"28e97e226f04e94e297303549528fa06b5abdec5c84334e0e0669247323082d8"}
Sep 15 11:41:01.024070 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.023966    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp" event={"ID":"8383a7d3-8b1e-429a-8dae-2dbb150a00fc","Type":"ContainerDied","Data":"a33000c949e8745932bd26aba75b2c4f633a4a87a57aee73c2f4d16e8fcb1291"}
Sep 15 11:41:01.024070 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.023800    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"
Sep 15 11:41:01.031184 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.031019    2562 scope.go:117] "RemoveContainer" containerID="e83e9d951d6f157e847035b5c15c25af1e6b1ead319cb443a8849444c4b2d3cf"
Sep 15 11:41:01.031529 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:41:01.031508    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e83e9d951d6f157e847035b5c15c25af1e6b1ead319cb443a8849444c4b2d3cf\": container with ID starting with e83e9d951d6f157e847035b5c15c25af1e6b1ead319cb443a8849444c4b2d3cf not found: ID does not exist" containerID="e83e9d951d6f157e847035b5c15c25af1e6b1ead319cb443a8849444c4b2d3cf"
Sep 15 11:41:01.031634 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.031534    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e83e9d951d6f157e847035b5c15c25af1e6b1ead319cb443a8849444c4b2d3cf"} err="failed to get container status \"e83e9d951d6f157e847035b5c15c25af1e6b1ead319cb443a8849444c4b2d3cf\": rpc error: code = NotFound desc = could not find container \"e83e9d951d6f157e847035b5c15c25af1e6b1ead319cb443a8849444c4b2d3cf\": container with ID starting with e83e9d951d6f157e847035b5c15c25af1e6b1ead319cb443a8849444c4b2d3cf not found: ID does not exist"
Sep 15 11:41:01.031634 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.031550    2562 scope.go:117] "RemoveContainer" containerID="2cae0125cfe3c1dc34176baa6f1adcb8184b70a2b048a67a256f81daee818afc"
Sep 15 11:41:01.039860 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.039651    2562 scope.go:117] "RemoveContainer" containerID="2cae0125cfe3c1dc34176baa6f1adcb8184b70a2b048a67a256f81daee818afc"
Sep 15 11:41:01.040168 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:41:01.039995    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cae0125cfe3c1dc34176baa6f1adcb8184b70a2b048a67a256f81daee818afc\": container with ID starting with 2cae0125cfe3c1dc34176baa6f1adcb8184b70a2b048a67a256f81daee818afc not found: ID does not exist" containerID="2cae0125cfe3c1dc34176baa6f1adcb8184b70a2b048a67a256f81daee818afc"
Sep 15 11:41:01.040168 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.040028    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cae0125cfe3c1dc34176baa6f1adcb8184b70a2b048a67a256f81daee818afc"} err="failed to get container status \"2cae0125cfe3c1dc34176baa6f1adcb8184b70a2b048a67a256f81daee818afc\": rpc error: code = NotFound desc = could not find container \"2cae0125cfe3c1dc34176baa6f1adcb8184b70a2b048a67a256f81daee818afc\": container with ID starting with 2cae0125cfe3c1dc34176baa6f1adcb8184b70a2b048a67a256f81daee818afc not found: ID does not exist"
Sep 15 11:41:01.040168 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.040050    2562 scope.go:117] "RemoveContainer" containerID="28e97e226f04e94e297303549528fa06b5abdec5c84334e0e0669247323082d8"
Sep 15 11:41:01.047777 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.047755    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx"]
Sep 15 11:41:01.059330 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.059309    2562 scope.go:117] "RemoveContainer" containerID="28e97e226f04e94e297303549528fa06b5abdec5c84334e0e0669247323082d8"
Sep 15 11:41:01.059625 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:41:01.059604    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e97e226f04e94e297303549528fa06b5abdec5c84334e0e0669247323082d8\": container with ID starting with 28e97e226f04e94e297303549528fa06b5abdec5c84334e0e0669247323082d8 not found: ID does not exist" containerID="28e97e226f04e94e297303549528fa06b5abdec5c84334e0e0669247323082d8"
Sep 15 11:41:01.059688 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.059630    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e97e226f04e94e297303549528fa06b5abdec5c84334e0e0669247323082d8"} err="failed to get container status \"28e97e226f04e94e297303549528fa06b5abdec5c84334e0e0669247323082d8\": rpc error: code = NotFound desc = could not find container \"28e97e226f04e94e297303549528fa06b5abdec5c84334e0e0669247323082d8\": container with ID starting with 28e97e226f04e94e297303549528fa06b5abdec5c84334e0e0669247323082d8 not found: ID does not exist"
Sep 15 11:41:01.060077 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.060042    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6f21996a-7f7f87fc77-stlrx"]
Sep 15 11:41:01.076673 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.076646    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"]
Sep 15 11:41:01.084898 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.084875    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6f21996a-74b79d5c6-lc4tp"]
Sep 15 11:41:01.102048 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.102025    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4"]
Sep 15 11:41:01.105722 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.105696    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6f21996a-555d757574-hn6j4"]
Sep 15 11:41:01.875985 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.875962    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18c700e8-a0fd-44ae-9232-69a88e161e90" path="/var/lib/kubelet/pods/18c700e8-a0fd-44ae-9232-69a88e161e90/volumes"
Sep 15 11:41:01.876376 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.876361    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26fb3833-f208-48c9-8f0b-1ec2c6ed81d8" path="/var/lib/kubelet/pods/26fb3833-f208-48c9-8f0b-1ec2c6ed81d8/volumes"
Sep 15 11:41:01.876707 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:01.876696    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8383a7d3-8b1e-429a-8dae-2dbb150a00fc" path="/var/lib/kubelet/pods/8383a7d3-8b1e-429a-8dae-2dbb150a00fc/volumes"
Sep 15 11:41:05.169485 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:05.169446    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:41:14.962518 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:14.962484    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"]
Sep 15 11:41:14.962900 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:14.962832    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26fb3833-f208-48c9-8f0b-1ec2c6ed81d8" containerName="payload-pre-processing"
Sep 15 11:41:14.962900 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:14.962843    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="26fb3833-f208-48c9-8f0b-1ec2c6ed81d8" containerName="payload-pre-processing"
Sep 15 11:41:14.962900 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:14.962854    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18c700e8-a0fd-44ae-9232-69a88e161e90" containerName="payload-processing"
Sep 15 11:41:14.962900 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:14.962860    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="18c700e8-a0fd-44ae-9232-69a88e161e90" containerName="payload-processing"
Sep 15 11:41:14.962900 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:14.962869    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8383a7d3-8b1e-429a-8dae-2dbb150a00fc" containerName="maas-api"
Sep 15 11:41:14.962900 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:14.962874    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8383a7d3-8b1e-429a-8dae-2dbb150a00fc" containerName="maas-api"
Sep 15 11:41:14.963118 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:14.962930    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="26fb3833-f208-48c9-8f0b-1ec2c6ed81d8" containerName="payload-pre-processing"
Sep 15 11:41:14.963118 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:14.962937    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8383a7d3-8b1e-429a-8dae-2dbb150a00fc" containerName="maas-api"
Sep 15 11:41:14.963118 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:14.962945    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="18c700e8-a0fd-44ae-9232-69a88e161e90" containerName="payload-processing"
Sep 15 11:41:14.967553 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:14.967536    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:14.971284 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:14.971258    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b8fbdd\"/\"default-dockercfg-62nm7\""
Sep 15 11:41:14.971409 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:14.971283    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b8fbdd\"/\"openshift-service-ca.crt\""
Sep 15 11:41:14.971409 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:14.971317    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b8fbdd\"/\"premium-w4-b8fbdd-kserve-self-signed-certs\""
Sep 15 11:41:14.971640 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:14.971619    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b8fbdd\"/\"kube-root-ca.crt\""
Sep 15 11:41:14.998307 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:14.998279    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"]
Sep 15 11:41:15.083177 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:15.083136    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-model-cache\") pod \"premium-w4-b8fbdd-kserve-858dc4666-97n5r\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") " pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:15.083363 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:15.083185    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-dshm\") pod \"premium-w4-b8fbdd-kserve-858dc4666-97n5r\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") " pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:15.083363 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:15.083205    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-home\") pod \"premium-w4-b8fbdd-kserve-858dc4666-97n5r\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") " pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:15.083363 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:15.083220    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-tmp-dir\") pod \"premium-w4-b8fbdd-kserve-858dc4666-97n5r\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") " pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:15.083363 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:15.083242    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqp7q\" (UniqueName: \"kubernetes.io/projected/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-kube-api-access-xqp7q\") pod \"premium-w4-b8fbdd-kserve-858dc4666-97n5r\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") " pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:15.083363 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:15.083269    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-tls-certs\") pod \"premium-w4-b8fbdd-kserve-858dc4666-97n5r\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") " pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:15.184179 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:15.184142    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-model-cache\") pod \"premium-w4-b8fbdd-kserve-858dc4666-97n5r\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") " pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:15.184389 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:15.184187    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-dshm\") pod \"premium-w4-b8fbdd-kserve-858dc4666-97n5r\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") " pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:15.184389 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:15.184213    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-home\") pod \"premium-w4-b8fbdd-kserve-858dc4666-97n5r\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") " pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:15.184389 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:15.184235    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-tmp-dir\") pod \"premium-w4-b8fbdd-kserve-858dc4666-97n5r\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") " pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:15.184389 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:15.184257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqp7q\" (UniqueName: \"kubernetes.io/projected/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-kube-api-access-xqp7q\") pod \"premium-w4-b8fbdd-kserve-858dc4666-97n5r\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") " pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:15.184389 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:15.184296    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-tls-certs\") pod \"premium-w4-b8fbdd-kserve-858dc4666-97n5r\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") " pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:15.184667 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:15.184634    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-home\") pod \"premium-w4-b8fbdd-kserve-858dc4666-97n5r\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") " pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:15.184725 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:15.184649    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-model-cache\") pod \"premium-w4-b8fbdd-kserve-858dc4666-97n5r\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") " pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:15.184784 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:15.184701    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-tmp-dir\") pod \"premium-w4-b8fbdd-kserve-858dc4666-97n5r\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") " pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:15.186499 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:15.186474    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-dshm\") pod \"premium-w4-b8fbdd-kserve-858dc4666-97n5r\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") " pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:15.186907 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:15.186873    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-tls-certs\") pod \"premium-w4-b8fbdd-kserve-858dc4666-97n5r\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") " pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:15.198758 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:15.198731    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqp7q\" (UniqueName: \"kubernetes.io/projected/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-kube-api-access-xqp7q\") pod \"premium-w4-b8fbdd-kserve-858dc4666-97n5r\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") " pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:15.282845 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:15.282755    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:15.569735 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:15.569706    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"]
Sep 15 11:41:15.571880 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:41:15.571854    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb2913b3_759c_4dcc_bcc4_fdb3311d1356.slice/crio-ba5e800ffcb3e541bffd5853371010115eebec8ffdad92885f28290210d2b5e7 WatchSource:0}: Error finding container ba5e800ffcb3e541bffd5853371010115eebec8ffdad92885f28290210d2b5e7: Status 404 returned error can't find the container with id ba5e800ffcb3e541bffd5853371010115eebec8ffdad92885f28290210d2b5e7
Sep 15 11:41:16.093147 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:16.093107    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r" event={"ID":"cb2913b3-759c-4dcc-bcc4-fdb3311d1356","Type":"ContainerStarted","Data":"d4f8ab42df5a97ee6785f886c9f1a49adf3519718a148dd18e2a84bc0cde6bc6"}
Sep 15 11:41:16.093147 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:16.093150    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r" event={"ID":"cb2913b3-759c-4dcc-bcc4-fdb3311d1356","Type":"ContainerStarted","Data":"ba5e800ffcb3e541bffd5853371010115eebec8ffdad92885f28290210d2b5e7"}
Sep 15 11:41:16.114124 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:16.114072    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r" podStartSLOduration=2.114042611 podStartE2EDuration="2.114042611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:41:16.112475674 +0000 UTC m=+974.774970855" watchObservedRunningTime="2026-09-15 11:41:16.114042611 +0000 UTC m=+974.776537792"
Sep 15 11:41:19.284798 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:19.284758    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54"]
Sep 15 11:41:19.288121 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:19.288103    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54"
Sep 15 11:41:19.290606 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:19.290583    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-a73e48\""
Sep 15 11:41:19.301043 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:19.301015    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54"]
Sep 15 11:41:19.326172 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:19.326140    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmqpq\" (UniqueName: \"kubernetes.io/projected/90b92ded-cd7b-4c0d-9b96-daeba9109ab9-kube-api-access-zmqpq\") pod \"payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54\" (UID: \"90b92ded-cd7b-4c0d-9b96-daeba9109ab9\") " pod="openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54"
Sep 15 11:41:19.326345 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:19.326194    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90b92ded-cd7b-4c0d-9b96-daeba9109ab9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54\" (UID: \"90b92ded-cd7b-4c0d-9b96-daeba9109ab9\") " pod="openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54"
Sep 15 11:41:19.427530 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:19.427493    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmqpq\" (UniqueName: \"kubernetes.io/projected/90b92ded-cd7b-4c0d-9b96-daeba9109ab9-kube-api-access-zmqpq\") pod \"payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54\" (UID: \"90b92ded-cd7b-4c0d-9b96-daeba9109ab9\") " pod="openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54"
Sep 15 11:41:19.427704 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:19.427562    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90b92ded-cd7b-4c0d-9b96-daeba9109ab9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54\" (UID: \"90b92ded-cd7b-4c0d-9b96-daeba9109ab9\") " pod="openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54"
Sep 15 11:41:19.428334 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:19.428309    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90b92ded-cd7b-4c0d-9b96-daeba9109ab9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54\" (UID: \"90b92ded-cd7b-4c0d-9b96-daeba9109ab9\") " pod="openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54"
Sep 15 11:41:19.437689 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:19.437664    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmqpq\" (UniqueName: \"kubernetes.io/projected/90b92ded-cd7b-4c0d-9b96-daeba9109ab9-kube-api-access-zmqpq\") pod \"payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54\" (UID: \"90b92ded-cd7b-4c0d-9b96-daeba9109ab9\") " pod="openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54"
Sep 15 11:41:19.600396 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:19.600363    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54"
Sep 15 11:41:19.875410 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:19.875387    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54"]
Sep 15 11:41:19.878162 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:41:19.878135    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b92ded_cd7b_4c0d_9b96_daeba9109ab9.slice/crio-3d887924f6e6f3e10887f98f553138615a53a5ad07091d0d4940973ed25e7af6 WatchSource:0}: Error finding container 3d887924f6e6f3e10887f98f553138615a53a5ad07091d0d4940973ed25e7af6: Status 404 returned error can't find the container with id 3d887924f6e6f3e10887f98f553138615a53a5ad07091d0d4940973ed25e7af6
Sep 15 11:41:20.113773 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:20.113734    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54" event={"ID":"90b92ded-cd7b-4c0d-9b96-daeba9109ab9","Type":"ContainerStarted","Data":"deba28ed15ea3f2f60c261c7680034b99e10f87bbbb023fcc70894d7d8d06b94"}
Sep 15 11:41:20.113987 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:20.113796    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54" event={"ID":"90b92ded-cd7b-4c0d-9b96-daeba9109ab9","Type":"ContainerStarted","Data":"3d887924f6e6f3e10887f98f553138615a53a5ad07091d0d4940973ed25e7af6"}
Sep 15 11:41:20.113987 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:20.113851    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54"
Sep 15 11:41:20.135581 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:20.135481    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54" podStartSLOduration=1.135464522 podStartE2EDuration="1.135464522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:41:20.132323086 +0000 UTC m=+978.794818287" watchObservedRunningTime="2026-09-15 11:41:20.135464522 +0000 UTC m=+978.797959702"
Sep 15 11:41:25.283678 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:25.283590    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:25.283678 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:25.283658    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:25.296018 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:25.295994    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:26.153358 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:26.153323    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:41:27.759909 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:27.759867    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:41:31.120986 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:41:31.120947    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54"
Sep 15 11:42:04.947416 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:04.947381    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:42:21.250525 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:21.250489    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:42:27.234805 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.234757    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"]
Sep 15 11:42:27.238751 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.238718    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:27.241505 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.241482    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-a73e48\"/\"kube-root-ca.crt\""
Sep 15 11:42:27.242189 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.242169    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-a73e48\"/\"openshift-service-ca.crt\""
Sep 15 11:42:27.242189 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.242177    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-a73e48\"/\"default-dockercfg-pnt2g\""
Sep 15 11:42:27.242474 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.242458    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-a73e48\"/\"explicit-model-a73e48-kserve-self-signed-certs\""
Sep 15 11:42:27.247726 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.247701    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"]
Sep 15 11:42:27.351888 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.351854    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-tmp-dir\") pod \"explicit-model-a73e48-kserve-5f5c6cf7bf-6db52\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") " pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:27.351888 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.351893    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4bbbc24-b972-4604-8c8c-4281e2dccf83-tls-certs\") pod \"explicit-model-a73e48-kserve-5f5c6cf7bf-6db52\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") " pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:27.352110 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.351999    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-model-cache\") pod \"explicit-model-a73e48-kserve-5f5c6cf7bf-6db52\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") " pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:27.352110 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.352030    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-dshm\") pod \"explicit-model-a73e48-kserve-5f5c6cf7bf-6db52\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") " pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:27.352110 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.352070    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2gb8\" (UniqueName: \"kubernetes.io/projected/e4bbbc24-b972-4604-8c8c-4281e2dccf83-kube-api-access-x2gb8\") pod \"explicit-model-a73e48-kserve-5f5c6cf7bf-6db52\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") " pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:27.352110 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.352107    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-home\") pod \"explicit-model-a73e48-kserve-5f5c6cf7bf-6db52\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") " pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:27.453305 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.453269    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-model-cache\") pod \"explicit-model-a73e48-kserve-5f5c6cf7bf-6db52\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") " pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:27.453305 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.453309    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-dshm\") pod \"explicit-model-a73e48-kserve-5f5c6cf7bf-6db52\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") " pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:27.453518 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.453420    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2gb8\" (UniqueName: \"kubernetes.io/projected/e4bbbc24-b972-4604-8c8c-4281e2dccf83-kube-api-access-x2gb8\") pod \"explicit-model-a73e48-kserve-5f5c6cf7bf-6db52\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") " pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:27.453518 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.453472    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-home\") pod \"explicit-model-a73e48-kserve-5f5c6cf7bf-6db52\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") " pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:27.453602 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.453572    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-tmp-dir\") pod \"explicit-model-a73e48-kserve-5f5c6cf7bf-6db52\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") " pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:27.453661 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.453619    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4bbbc24-b972-4604-8c8c-4281e2dccf83-tls-certs\") pod \"explicit-model-a73e48-kserve-5f5c6cf7bf-6db52\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") " pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:27.453935 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.453911    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-model-cache\") pod \"explicit-model-a73e48-kserve-5f5c6cf7bf-6db52\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") " pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:27.454043 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.453940    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-home\") pod \"explicit-model-a73e48-kserve-5f5c6cf7bf-6db52\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") " pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:27.454043 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.453987    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-tmp-dir\") pod \"explicit-model-a73e48-kserve-5f5c6cf7bf-6db52\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") " pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:27.455652 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.455627    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-dshm\") pod \"explicit-model-a73e48-kserve-5f5c6cf7bf-6db52\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") " pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:27.456037 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.456020    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4bbbc24-b972-4604-8c8c-4281e2dccf83-tls-certs\") pod \"explicit-model-a73e48-kserve-5f5c6cf7bf-6db52\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") " pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:27.461521 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.461499    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2gb8\" (UniqueName: \"kubernetes.io/projected/e4bbbc24-b972-4604-8c8c-4281e2dccf83-kube-api-access-x2gb8\") pod \"explicit-model-a73e48-kserve-5f5c6cf7bf-6db52\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") " pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:27.549604 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.549517    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:27.823256 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:27.823229    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"]
Sep 15 11:42:27.825366 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:42:27.825327    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4bbbc24_b972_4604_8c8c_4281e2dccf83.slice/crio-f4f1daaf0398b59b63242b4aae148eaf25e801b5b5e847ce473438b7099e390d WatchSource:0}: Error finding container f4f1daaf0398b59b63242b4aae148eaf25e801b5b5e847ce473438b7099e390d: Status 404 returned error can't find the container with id f4f1daaf0398b59b63242b4aae148eaf25e801b5b5e847ce473438b7099e390d
Sep 15 11:42:28.432751 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:28.432714    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52" event={"ID":"e4bbbc24-b972-4604-8c8c-4281e2dccf83","Type":"ContainerStarted","Data":"ab599d075597748257d6afe405dd50b6373262accc5aefc44157eac57337da02"}
Sep 15 11:42:28.432751 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:28.432759    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52" event={"ID":"e4bbbc24-b972-4604-8c8c-4281e2dccf83","Type":"ContainerStarted","Data":"f4f1daaf0398b59b63242b4aae148eaf25e801b5b5e847ce473438b7099e390d"}
Sep 15 11:42:28.451717 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:28.451658    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52" podStartSLOduration=1.4516415999999999 podStartE2EDuration="1.4516416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:42:28.449409269 +0000 UTC m=+1047.111904449" watchObservedRunningTime="2026-09-15 11:42:28.4516416 +0000 UTC m=+1047.114136822"
Sep 15 11:42:37.550237 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:37.550200    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:37.550655 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:37.550303    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:37.562922 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:37.562889    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:38.497678 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:38.497636    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:42:40.791871 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:40.791835    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:42:47.980203 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:47.980153    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs"]
Sep 15 11:42:47.983654 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:47.983635    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs"
Sep 15 11:42:47.985837 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:47.985807    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5f7ca109-dockercfg-xksdw\""
Sep 15 11:42:47.986123 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:47.986103    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5f7ca109\""
Sep 15 11:42:47.997025 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:47.996999    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs"]
Sep 15 11:42:48.000613 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:48.000588    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp"]
Sep 15 11:42:48.004556 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:48.004532    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp"
Sep 15 11:42:48.014925 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:48.014905    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp"]
Sep 15 11:42:48.043217 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:48.043181    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldqlv\" (UniqueName: \"kubernetes.io/projected/272b6300-1b15-4e4d-ad30-0be24fe5d26a-kube-api-access-ldqlv\") pod \"payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs\" (UID: \"272b6300-1b15-4e4d-ad30-0be24fe5d26a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs"
Sep 15 11:42:48.043379 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:48.043254    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272b6300-1b15-4e4d-ad30-0be24fe5d26a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs\" (UID: \"272b6300-1b15-4e4d-ad30-0be24fe5d26a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs"
Sep 15 11:42:48.043379 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:48.043272    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnvjk\" (UniqueName: \"kubernetes.io/projected/f937c9ec-7414-4ba2-8f39-73bfbca146ec-kube-api-access-gnvjk\") pod \"payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp\" (UID: \"f937c9ec-7414-4ba2-8f39-73bfbca146ec\") " pod="openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp"
Sep 15 11:42:48.043379 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:48.043295    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f937c9ec-7414-4ba2-8f39-73bfbca146ec-plugins-config-volume\") pod \"payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp\" (UID: \"f937c9ec-7414-4ba2-8f39-73bfbca146ec\") " pod="openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp"
Sep 15 11:42:48.144521 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:48.144483    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272b6300-1b15-4e4d-ad30-0be24fe5d26a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs\" (UID: \"272b6300-1b15-4e4d-ad30-0be24fe5d26a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs"
Sep 15 11:42:48.144521 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:48.144523    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnvjk\" (UniqueName: \"kubernetes.io/projected/f937c9ec-7414-4ba2-8f39-73bfbca146ec-kube-api-access-gnvjk\") pod \"payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp\" (UID: \"f937c9ec-7414-4ba2-8f39-73bfbca146ec\") " pod="openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp"
Sep 15 11:42:48.144766 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:48.144557    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f937c9ec-7414-4ba2-8f39-73bfbca146ec-plugins-config-volume\") pod \"payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp\" (UID: \"f937c9ec-7414-4ba2-8f39-73bfbca146ec\") " pod="openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp"
Sep 15 11:42:48.144766 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:48.144736    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldqlv\" (UniqueName: \"kubernetes.io/projected/272b6300-1b15-4e4d-ad30-0be24fe5d26a-kube-api-access-ldqlv\") pod \"payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs\" (UID: \"272b6300-1b15-4e4d-ad30-0be24fe5d26a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs"
Sep 15 11:42:48.145241 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:48.145222    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272b6300-1b15-4e4d-ad30-0be24fe5d26a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs\" (UID: \"272b6300-1b15-4e4d-ad30-0be24fe5d26a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs"
Sep 15 11:42:48.145333 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:48.145315    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f937c9ec-7414-4ba2-8f39-73bfbca146ec-plugins-config-volume\") pod \"payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp\" (UID: \"f937c9ec-7414-4ba2-8f39-73bfbca146ec\") " pod="openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp"
Sep 15 11:42:48.153295 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:48.153272    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldqlv\" (UniqueName: \"kubernetes.io/projected/272b6300-1b15-4e4d-ad30-0be24fe5d26a-kube-api-access-ldqlv\") pod \"payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs\" (UID: \"272b6300-1b15-4e4d-ad30-0be24fe5d26a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs"
Sep 15 11:42:48.153412 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:48.153311    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnvjk\" (UniqueName: \"kubernetes.io/projected/f937c9ec-7414-4ba2-8f39-73bfbca146ec-kube-api-access-gnvjk\") pod \"payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp\" (UID: \"f937c9ec-7414-4ba2-8f39-73bfbca146ec\") " pod="openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp"
Sep 15 11:42:48.294530 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:48.294424    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs"
Sep 15 11:42:48.315125 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:48.315098    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp"
Sep 15 11:42:48.569426 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:48.569345    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs"]
Sep 15 11:42:48.574287 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:42:48.574258    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272b6300_1b15_4e4d_ad30_0be24fe5d26a.slice/crio-4c8382d8d4b322e703bcc98c2151e655d691f640d64c90add30f8878f8227e5b WatchSource:0}: Error finding container 4c8382d8d4b322e703bcc98c2151e655d691f640d64c90add30f8878f8227e5b: Status 404 returned error can't find the container with id 4c8382d8d4b322e703bcc98c2151e655d691f640d64c90add30f8878f8227e5b
Sep 15 11:42:48.600601 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:48.600577    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp"]
Sep 15 11:42:48.609191 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:42:48.609159    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf937c9ec_7414_4ba2_8f39_73bfbca146ec.slice/crio-778c7bad776e908a981b5d8928be59caac8b4909fea39e327acdadb8d44bca88 WatchSource:0}: Error finding container 778c7bad776e908a981b5d8928be59caac8b4909fea39e327acdadb8d44bca88: Status 404 returned error can't find the container with id 778c7bad776e908a981b5d8928be59caac8b4909fea39e327acdadb8d44bca88
Sep 15 11:42:49.537238 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:49.537202    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp" event={"ID":"f937c9ec-7414-4ba2-8f39-73bfbca146ec","Type":"ContainerStarted","Data":"9ff79a2c392b2b4a40e6ddd241c1ad8ee68c27af3139cb5231b49eef7c3ae84c"}
Sep 15 11:42:49.537726 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:49.537260    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp" event={"ID":"f937c9ec-7414-4ba2-8f39-73bfbca146ec","Type":"ContainerStarted","Data":"778c7bad776e908a981b5d8928be59caac8b4909fea39e327acdadb8d44bca88"}
Sep 15 11:42:49.537726 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:49.537412    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp"
Sep 15 11:42:49.538753 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:49.538729    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs" event={"ID":"272b6300-1b15-4e4d-ad30-0be24fe5d26a","Type":"ContainerStarted","Data":"7c4228130ed1520aefe072d80dc88a033c9d95f595a5c351eb7282f90ab63244"}
Sep 15 11:42:49.538852 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:49.538775    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs" event={"ID":"272b6300-1b15-4e4d-ad30-0be24fe5d26a","Type":"ContainerStarted","Data":"4c8382d8d4b322e703bcc98c2151e655d691f640d64c90add30f8878f8227e5b"}
Sep 15 11:42:49.538852 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:49.538815    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs"
Sep 15 11:42:49.554691 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:49.554641    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp" podStartSLOduration=2.554623521 podStartE2EDuration="2.554623521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:42:49.553544148 +0000 UTC m=+1068.216039336" watchObservedRunningTime="2026-09-15 11:42:49.554623521 +0000 UTC m=+1068.217118705"
Sep 15 11:42:49.571457 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:49.571408    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs" podStartSLOduration=2.571389844 podStartE2EDuration="2.571389844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:42:49.569874432 +0000 UTC m=+1068.232369633" watchObservedRunningTime="2026-09-15 11:42:49.571389844 +0000 UTC m=+1068.233885031"
Sep 15 11:42:56.943293 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:42:56.943209    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"]
Sep 15 11:43:00.544863 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:00.544821    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp"
Sep 15 11:43:00.545379 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:00.545109    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs"
Sep 15 11:43:07.857405 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:07.857356    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54"]
Sep 15 11:43:07.858771 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:07.858719    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54" podUID="90b92ded-cd7b-4c0d-9b96-daeba9109ab9" containerName="payload-processing" containerID="cri-o://deba28ed15ea3f2f60c261c7680034b99e10f87bbbb023fcc70894d7d8d06b94" gracePeriod=30
Sep 15 11:43:08.120143 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:08.120079    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54"
Sep 15 11:43:08.229206 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:08.229174    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90b92ded-cd7b-4c0d-9b96-daeba9109ab9-plugins-config-volume\") pod \"90b92ded-cd7b-4c0d-9b96-daeba9109ab9\" (UID: \"90b92ded-cd7b-4c0d-9b96-daeba9109ab9\") "
Sep 15 11:43:08.229390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:08.229281    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmqpq\" (UniqueName: \"kubernetes.io/projected/90b92ded-cd7b-4c0d-9b96-daeba9109ab9-kube-api-access-zmqpq\") pod \"90b92ded-cd7b-4c0d-9b96-daeba9109ab9\" (UID: \"90b92ded-cd7b-4c0d-9b96-daeba9109ab9\") "
Sep 15 11:43:08.229538 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:08.229507    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b92ded-cd7b-4c0d-9b96-daeba9109ab9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "90b92ded-cd7b-4c0d-9b96-daeba9109ab9" (UID: "90b92ded-cd7b-4c0d-9b96-daeba9109ab9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:43:08.231386 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:08.231359    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b92ded-cd7b-4c0d-9b96-daeba9109ab9-kube-api-access-zmqpq" (OuterVolumeSpecName: "kube-api-access-zmqpq") pod "90b92ded-cd7b-4c0d-9b96-daeba9109ab9" (UID: "90b92ded-cd7b-4c0d-9b96-daeba9109ab9"). InnerVolumeSpecName "kube-api-access-zmqpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:43:08.330727 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:08.330691    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90b92ded-cd7b-4c0d-9b96-daeba9109ab9-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:43:08.330727 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:08.330719    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmqpq\" (UniqueName: \"kubernetes.io/projected/90b92ded-cd7b-4c0d-9b96-daeba9109ab9-kube-api-access-zmqpq\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:43:08.633670 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:08.633633    2562 generic.go:356] "Generic (PLEG): container finished" podID="90b92ded-cd7b-4c0d-9b96-daeba9109ab9" containerID="deba28ed15ea3f2f60c261c7680034b99e10f87bbbb023fcc70894d7d8d06b94" exitCode=0
Sep 15 11:43:08.633874 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:08.633678    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54"
Sep 15 11:43:08.633874 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:08.633694    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54" event={"ID":"90b92ded-cd7b-4c0d-9b96-daeba9109ab9","Type":"ContainerDied","Data":"deba28ed15ea3f2f60c261c7680034b99e10f87bbbb023fcc70894d7d8d06b94"}
Sep 15 11:43:08.633874 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:08.633735    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54" event={"ID":"90b92ded-cd7b-4c0d-9b96-daeba9109ab9","Type":"ContainerDied","Data":"3d887924f6e6f3e10887f98f553138615a53a5ad07091d0d4940973ed25e7af6"}
Sep 15 11:43:08.633874 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:08.633756    2562 scope.go:117] "RemoveContainer" containerID="deba28ed15ea3f2f60c261c7680034b99e10f87bbbb023fcc70894d7d8d06b94"
Sep 15 11:43:08.643225 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:08.643205    2562 scope.go:117] "RemoveContainer" containerID="deba28ed15ea3f2f60c261c7680034b99e10f87bbbb023fcc70894d7d8d06b94"
Sep 15 11:43:08.643534 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:43:08.643517    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deba28ed15ea3f2f60c261c7680034b99e10f87bbbb023fcc70894d7d8d06b94\": container with ID starting with deba28ed15ea3f2f60c261c7680034b99e10f87bbbb023fcc70894d7d8d06b94 not found: ID does not exist" containerID="deba28ed15ea3f2f60c261c7680034b99e10f87bbbb023fcc70894d7d8d06b94"
Sep 15 11:43:08.643574 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:08.643543    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deba28ed15ea3f2f60c261c7680034b99e10f87bbbb023fcc70894d7d8d06b94"} err="failed to get container status \"deba28ed15ea3f2f60c261c7680034b99e10f87bbbb023fcc70894d7d8d06b94\": rpc error: code = NotFound desc = could not find container \"deba28ed15ea3f2f60c261c7680034b99e10f87bbbb023fcc70894d7d8d06b94\": container with ID starting with deba28ed15ea3f2f60c261c7680034b99e10f87bbbb023fcc70894d7d8d06b94 not found: ID does not exist"
Sep 15 11:43:08.687005 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:08.686970    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54"]
Sep 15 11:43:08.709082 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:08.709035    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a73e48-694cd5bbd9-wbw54"]
Sep 15 11:43:09.873996 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:09.873965    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90b92ded-cd7b-4c0d-9b96-daeba9109ab9" path="/var/lib/kubelet/pods/90b92ded-cd7b-4c0d-9b96-daeba9109ab9/volumes"
Sep 15 11:43:11.962670 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:11.962634    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52" podUID="e4bbbc24-b972-4604-8c8c-4281e2dccf83" containerName="main" containerID="cri-o://ab599d075597748257d6afe405dd50b6373262accc5aefc44157eac57337da02" gracePeriod=45
Sep 15 11:43:12.222944 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.222873    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:43:12.365358 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.365321    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-home\") pod \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") "
Sep 15 11:43:12.365566 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.365371    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-model-cache\") pod \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") "
Sep 15 11:43:12.365566 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.365406    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2gb8\" (UniqueName: \"kubernetes.io/projected/e4bbbc24-b972-4604-8c8c-4281e2dccf83-kube-api-access-x2gb8\") pod \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") "
Sep 15 11:43:12.365566 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.365465    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-tmp-dir\") pod \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") "
Sep 15 11:43:12.365566 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.365508    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4bbbc24-b972-4604-8c8c-4281e2dccf83-tls-certs\") pod \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") "
Sep 15 11:43:12.365783 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.365650    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4bbbc24-b972-4604-8c8c-4281e2dccf83" (UID: "e4bbbc24-b972-4604-8c8c-4281e2dccf83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:43:12.365783 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.365674    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-dshm\") pod \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\" (UID: \"e4bbbc24-b972-4604-8c8c-4281e2dccf83\") "
Sep 15 11:43:12.365783 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.365679    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-home" (OuterVolumeSpecName: "home") pod "e4bbbc24-b972-4604-8c8c-4281e2dccf83" (UID: "e4bbbc24-b972-4604-8c8c-4281e2dccf83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:43:12.365783 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.365747    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4bbbc24-b972-4604-8c8c-4281e2dccf83" (UID: "e4bbbc24-b972-4604-8c8c-4281e2dccf83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:43:12.366096 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.366045    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:43:12.366096 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.366096    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:43:12.366244 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.366106    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:43:12.368017 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.367988    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4bbbc24-b972-4604-8c8c-4281e2dccf83-kube-api-access-x2gb8" (OuterVolumeSpecName: "kube-api-access-x2gb8") pod "e4bbbc24-b972-4604-8c8c-4281e2dccf83" (UID: "e4bbbc24-b972-4604-8c8c-4281e2dccf83"). InnerVolumeSpecName "kube-api-access-x2gb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:43:12.368176 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.367988    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-dshm" (OuterVolumeSpecName: "dshm") pod "e4bbbc24-b972-4604-8c8c-4281e2dccf83" (UID: "e4bbbc24-b972-4604-8c8c-4281e2dccf83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:43:12.368176 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.368012    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4bbbc24-b972-4604-8c8c-4281e2dccf83-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4bbbc24-b972-4604-8c8c-4281e2dccf83" (UID: "e4bbbc24-b972-4604-8c8c-4281e2dccf83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:43:12.466701 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.466660    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4bbbc24-b972-4604-8c8c-4281e2dccf83-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:43:12.466701 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.466692    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4bbbc24-b972-4604-8c8c-4281e2dccf83-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:43:12.466701 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.466703    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2gb8\" (UniqueName: \"kubernetes.io/projected/e4bbbc24-b972-4604-8c8c-4281e2dccf83-kube-api-access-x2gb8\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:43:12.655278 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.655244    2562 generic.go:356] "Generic (PLEG): container finished" podID="e4bbbc24-b972-4604-8c8c-4281e2dccf83" containerID="ab599d075597748257d6afe405dd50b6373262accc5aefc44157eac57337da02" exitCode=0
Sep 15 11:43:12.655459 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.655287    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"
Sep 15 11:43:12.655459 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.655299    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52" event={"ID":"e4bbbc24-b972-4604-8c8c-4281e2dccf83","Type":"ContainerDied","Data":"ab599d075597748257d6afe405dd50b6373262accc5aefc44157eac57337da02"}
Sep 15 11:43:12.655459 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.655343    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52" event={"ID":"e4bbbc24-b972-4604-8c8c-4281e2dccf83","Type":"ContainerDied","Data":"f4f1daaf0398b59b63242b4aae148eaf25e801b5b5e847ce473438b7099e390d"}
Sep 15 11:43:12.655459 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.655361    2562 scope.go:117] "RemoveContainer" containerID="ab599d075597748257d6afe405dd50b6373262accc5aefc44157eac57337da02"
Sep 15 11:43:12.664945 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.664927    2562 scope.go:117] "RemoveContainer" containerID="ab599d075597748257d6afe405dd50b6373262accc5aefc44157eac57337da02"
Sep 15 11:43:12.665288 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:43:12.665268    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab599d075597748257d6afe405dd50b6373262accc5aefc44157eac57337da02\": container with ID starting with ab599d075597748257d6afe405dd50b6373262accc5aefc44157eac57337da02 not found: ID does not exist" containerID="ab599d075597748257d6afe405dd50b6373262accc5aefc44157eac57337da02"
Sep 15 11:43:12.665383 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.665293    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab599d075597748257d6afe405dd50b6373262accc5aefc44157eac57337da02"} err="failed to get container status \"ab599d075597748257d6afe405dd50b6373262accc5aefc44157eac57337da02\": rpc error: code = NotFound desc = could not find container \"ab599d075597748257d6afe405dd50b6373262accc5aefc44157eac57337da02\": container with ID starting with ab599d075597748257d6afe405dd50b6373262accc5aefc44157eac57337da02 not found: ID does not exist"
Sep 15 11:43:12.680417 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.680393    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"]
Sep 15 11:43:12.687900 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:12.687877    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-a73e48/explicit-model-a73e48-kserve-5f5c6cf7bf-6db52"]
Sep 15 11:43:13.873631 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:13.873591    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4bbbc24-b972-4604-8c8c-4281e2dccf83" path="/var/lib/kubelet/pods/e4bbbc24-b972-4604-8c8c-4281e2dccf83/volumes"
Sep 15 11:43:15.467064 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:15.467026    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:43:24.164408 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.164372    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp"]
Sep 15 11:43:24.164841 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.164587    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp" podUID="f937c9ec-7414-4ba2-8f39-73bfbca146ec" containerName="payload-processing" containerID="cri-o://9ff79a2c392b2b4a40e6ddd241c1ad8ee68c27af3139cb5231b49eef7c3ae84c" gracePeriod=30
Sep 15 11:43:24.168881 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.168854    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs"]
Sep 15 11:43:24.169670 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.169051    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs" podUID="272b6300-1b15-4e4d-ad30-0be24fe5d26a" containerName="payload-pre-processing" containerID="cri-o://7c4228130ed1520aefe072d80dc88a033c9d95f595a5c351eb7282f90ab63244" gracePeriod=30
Sep 15 11:43:24.446108 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.446086    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs"
Sep 15 11:43:24.448617 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.448599    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp"
Sep 15 11:43:24.576651 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.576617    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272b6300-1b15-4e4d-ad30-0be24fe5d26a-plugins-config-volume\") pod \"272b6300-1b15-4e4d-ad30-0be24fe5d26a\" (UID: \"272b6300-1b15-4e4d-ad30-0be24fe5d26a\") "
Sep 15 11:43:24.576846 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.576673    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldqlv\" (UniqueName: \"kubernetes.io/projected/272b6300-1b15-4e4d-ad30-0be24fe5d26a-kube-api-access-ldqlv\") pod \"272b6300-1b15-4e4d-ad30-0be24fe5d26a\" (UID: \"272b6300-1b15-4e4d-ad30-0be24fe5d26a\") "
Sep 15 11:43:24.576846 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.576713    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnvjk\" (UniqueName: \"kubernetes.io/projected/f937c9ec-7414-4ba2-8f39-73bfbca146ec-kube-api-access-gnvjk\") pod \"f937c9ec-7414-4ba2-8f39-73bfbca146ec\" (UID: \"f937c9ec-7414-4ba2-8f39-73bfbca146ec\") "
Sep 15 11:43:24.576846 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.576740    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f937c9ec-7414-4ba2-8f39-73bfbca146ec-plugins-config-volume\") pod \"f937c9ec-7414-4ba2-8f39-73bfbca146ec\" (UID: \"f937c9ec-7414-4ba2-8f39-73bfbca146ec\") "
Sep 15 11:43:24.577027 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.576947    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272b6300-1b15-4e4d-ad30-0be24fe5d26a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "272b6300-1b15-4e4d-ad30-0be24fe5d26a" (UID: "272b6300-1b15-4e4d-ad30-0be24fe5d26a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:43:24.577234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.577207    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f937c9ec-7414-4ba2-8f39-73bfbca146ec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f937c9ec-7414-4ba2-8f39-73bfbca146ec" (UID: "f937c9ec-7414-4ba2-8f39-73bfbca146ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:43:24.578897 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.578876    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f937c9ec-7414-4ba2-8f39-73bfbca146ec-kube-api-access-gnvjk" (OuterVolumeSpecName: "kube-api-access-gnvjk") pod "f937c9ec-7414-4ba2-8f39-73bfbca146ec" (UID: "f937c9ec-7414-4ba2-8f39-73bfbca146ec"). InnerVolumeSpecName "kube-api-access-gnvjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:43:24.578962 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.578899    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272b6300-1b15-4e4d-ad30-0be24fe5d26a-kube-api-access-ldqlv" (OuterVolumeSpecName: "kube-api-access-ldqlv") pod "272b6300-1b15-4e4d-ad30-0be24fe5d26a" (UID: "272b6300-1b15-4e4d-ad30-0be24fe5d26a"). InnerVolumeSpecName "kube-api-access-ldqlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:43:24.677635 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.677552    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldqlv\" (UniqueName: \"kubernetes.io/projected/272b6300-1b15-4e4d-ad30-0be24fe5d26a-kube-api-access-ldqlv\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:43:24.677635 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.677580    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnvjk\" (UniqueName: \"kubernetes.io/projected/f937c9ec-7414-4ba2-8f39-73bfbca146ec-kube-api-access-gnvjk\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:43:24.677635 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.677589    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f937c9ec-7414-4ba2-8f39-73bfbca146ec-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:43:24.677635 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.677599    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272b6300-1b15-4e4d-ad30-0be24fe5d26a-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:43:24.721213 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.721170    2562 generic.go:356] "Generic (PLEG): container finished" podID="f937c9ec-7414-4ba2-8f39-73bfbca146ec" containerID="9ff79a2c392b2b4a40e6ddd241c1ad8ee68c27af3139cb5231b49eef7c3ae84c" exitCode=0
Sep 15 11:43:24.721400 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.721228    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp"
Sep 15 11:43:24.721400 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.721250    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp" event={"ID":"f937c9ec-7414-4ba2-8f39-73bfbca146ec","Type":"ContainerDied","Data":"9ff79a2c392b2b4a40e6ddd241c1ad8ee68c27af3139cb5231b49eef7c3ae84c"}
Sep 15 11:43:24.721400 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.721338    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp" event={"ID":"f937c9ec-7414-4ba2-8f39-73bfbca146ec","Type":"ContainerDied","Data":"778c7bad776e908a981b5d8928be59caac8b4909fea39e327acdadb8d44bca88"}
Sep 15 11:43:24.721400 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.721370    2562 scope.go:117] "RemoveContainer" containerID="9ff79a2c392b2b4a40e6ddd241c1ad8ee68c27af3139cb5231b49eef7c3ae84c"
Sep 15 11:43:24.722734 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.722716    2562 generic.go:356] "Generic (PLEG): container finished" podID="272b6300-1b15-4e4d-ad30-0be24fe5d26a" containerID="7c4228130ed1520aefe072d80dc88a033c9d95f595a5c351eb7282f90ab63244" exitCode=0
Sep 15 11:43:24.722820 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.722786    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs"
Sep 15 11:43:24.722820 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.722794    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs" event={"ID":"272b6300-1b15-4e4d-ad30-0be24fe5d26a","Type":"ContainerDied","Data":"7c4228130ed1520aefe072d80dc88a033c9d95f595a5c351eb7282f90ab63244"}
Sep 15 11:43:24.722921 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.722843    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs" event={"ID":"272b6300-1b15-4e4d-ad30-0be24fe5d26a","Type":"ContainerDied","Data":"4c8382d8d4b322e703bcc98c2151e655d691f640d64c90add30f8878f8227e5b"}
Sep 15 11:43:24.730742 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.730722    2562 scope.go:117] "RemoveContainer" containerID="9ff79a2c392b2b4a40e6ddd241c1ad8ee68c27af3139cb5231b49eef7c3ae84c"
Sep 15 11:43:24.731018 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:43:24.731000    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff79a2c392b2b4a40e6ddd241c1ad8ee68c27af3139cb5231b49eef7c3ae84c\": container with ID starting with 9ff79a2c392b2b4a40e6ddd241c1ad8ee68c27af3139cb5231b49eef7c3ae84c not found: ID does not exist" containerID="9ff79a2c392b2b4a40e6ddd241c1ad8ee68c27af3139cb5231b49eef7c3ae84c"
Sep 15 11:43:24.731103 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.731026    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff79a2c392b2b4a40e6ddd241c1ad8ee68c27af3139cb5231b49eef7c3ae84c"} err="failed to get container status \"9ff79a2c392b2b4a40e6ddd241c1ad8ee68c27af3139cb5231b49eef7c3ae84c\": rpc error: code = NotFound desc = could not find container \"9ff79a2c392b2b4a40e6ddd241c1ad8ee68c27af3139cb5231b49eef7c3ae84c\": container with ID starting with 9ff79a2c392b2b4a40e6ddd241c1ad8ee68c27af3139cb5231b49eef7c3ae84c not found: ID does not exist"
Sep 15 11:43:24.731103 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.731042    2562 scope.go:117] "RemoveContainer" containerID="7c4228130ed1520aefe072d80dc88a033c9d95f595a5c351eb7282f90ab63244"
Sep 15 11:43:24.739477 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.739459    2562 scope.go:117] "RemoveContainer" containerID="7c4228130ed1520aefe072d80dc88a033c9d95f595a5c351eb7282f90ab63244"
Sep 15 11:43:24.739751 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:43:24.739732    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c4228130ed1520aefe072d80dc88a033c9d95f595a5c351eb7282f90ab63244\": container with ID starting with 7c4228130ed1520aefe072d80dc88a033c9d95f595a5c351eb7282f90ab63244 not found: ID does not exist" containerID="7c4228130ed1520aefe072d80dc88a033c9d95f595a5c351eb7282f90ab63244"
Sep 15 11:43:24.739817 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.739757    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c4228130ed1520aefe072d80dc88a033c9d95f595a5c351eb7282f90ab63244"} err="failed to get container status \"7c4228130ed1520aefe072d80dc88a033c9d95f595a5c351eb7282f90ab63244\": rpc error: code = NotFound desc = could not find container \"7c4228130ed1520aefe072d80dc88a033c9d95f595a5c351eb7282f90ab63244\": container with ID starting with 7c4228130ed1520aefe072d80dc88a033c9d95f595a5c351eb7282f90ab63244 not found: ID does not exist"
Sep 15 11:43:24.747722 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.747698    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs"]
Sep 15 11:43:24.754128 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.754108    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5f7ca109-6dbfcdcdd6-6tzvs"]
Sep 15 11:43:24.765006 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.764985    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp"]
Sep 15 11:43:24.771584 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:24.771563    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5f7ca109-f8b9454c7-l6ddp"]
Sep 15 11:43:25.873852 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:25.873817    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="272b6300-1b15-4e4d-ad30-0be24fe5d26a" path="/var/lib/kubelet/pods/272b6300-1b15-4e4d-ad30-0be24fe5d26a/volumes"
Sep 15 11:43:25.874285 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:25.874199    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f937c9ec-7414-4ba2-8f39-73bfbca146ec" path="/var/lib/kubelet/pods/f937c9ec-7414-4ba2-8f39-73bfbca146ec/volumes"
Sep 15 11:43:48.812049 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:48.812007    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:43:51.985512 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:51.985476    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"]
Sep 15 11:43:51.986008 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:51.985990    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f937c9ec-7414-4ba2-8f39-73bfbca146ec" containerName="payload-processing"
Sep 15 11:43:51.986084 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:51.986011    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f937c9ec-7414-4ba2-8f39-73bfbca146ec" containerName="payload-processing"
Sep 15 11:43:51.986084 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:51.986033    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272b6300-1b15-4e4d-ad30-0be24fe5d26a" containerName="payload-pre-processing"
Sep 15 11:43:51.986084 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:51.986042    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="272b6300-1b15-4e4d-ad30-0be24fe5d26a" containerName="payload-pre-processing"
Sep 15 11:43:51.986296 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:51.986086    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90b92ded-cd7b-4c0d-9b96-daeba9109ab9" containerName="payload-processing"
Sep 15 11:43:51.986296 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:51.986096    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="90b92ded-cd7b-4c0d-9b96-daeba9109ab9" containerName="payload-processing"
Sep 15 11:43:51.986296 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:51.986117    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4bbbc24-b972-4604-8c8c-4281e2dccf83" containerName="main"
Sep 15 11:43:51.986296 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:51.986125    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4bbbc24-b972-4604-8c8c-4281e2dccf83" containerName="main"
Sep 15 11:43:51.986296 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:51.986241    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="272b6300-1b15-4e4d-ad30-0be24fe5d26a" containerName="payload-pre-processing"
Sep 15 11:43:51.986296 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:51.986257    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4bbbc24-b972-4604-8c8c-4281e2dccf83" containerName="main"
Sep 15 11:43:51.986296 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:51.986269    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="90b92ded-cd7b-4c0d-9b96-daeba9109ab9" containerName="payload-processing"
Sep 15 11:43:51.986296 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:51.986280    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f937c9ec-7414-4ba2-8f39-73bfbca146ec" containerName="payload-processing"
Sep 15 11:43:51.989836 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:51.989813    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"
Sep 15 11:43:51.992178 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:51.992156    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-5a8d6f-serving-cert\""
Sep 15 11:43:52.005500 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.005477    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"]
Sep 15 11:43:52.123306 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.123275    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2\" (UID: \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"
Sep 15 11:43:52.123306 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.123307    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2\" (UID: \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"
Sep 15 11:43:52.123536 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.123340    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2\" (UID: \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"
Sep 15 11:43:52.123536 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.123375    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqzcc\" (UniqueName: \"kubernetes.io/projected/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-kube-api-access-mqzcc\") pod \"maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2\" (UID: \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"
Sep 15 11:43:52.224905 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.224855    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2\" (UID: \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"
Sep 15 11:43:52.225108 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.224912    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2\" (UID: \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"
Sep 15 11:43:52.225108 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.224961    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2\" (UID: \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"
Sep 15 11:43:52.225108 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.224994    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqzcc\" (UniqueName: \"kubernetes.io/projected/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-kube-api-access-mqzcc\") pod \"maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2\" (UID: \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"
Sep 15 11:43:52.225108 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:43:52.225021    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-serving-cert: secret "maas-api-e2e-shared-a-w2-5a8d6f-serving-cert" not found
Sep 15 11:43:52.225323 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:43:52.225147    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-maas-api-tls podName:6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b nodeName:}" failed. No retries permitted until 2026-09-15 11:43:52.725124258 +0000 UTC m=+1131.387619439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-maas-api-tls") pod "maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2" (UID: "6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b") : secret "maas-api-e2e-shared-a-w2-5a8d6f-serving-cert" not found
Sep 15 11:43:52.225721 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.225699    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2\" (UID: \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"
Sep 15 11:43:52.227489 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.227471    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2\" (UID: \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"
Sep 15 11:43:52.237081 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.237004    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqzcc\" (UniqueName: \"kubernetes.io/projected/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-kube-api-access-mqzcc\") pod \"maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2\" (UID: \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"
Sep 15 11:43:52.602905 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.602864    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l"]
Sep 15 11:43:52.606669 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.606646    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l"
Sep 15 11:43:52.609148 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.609122    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-5a8d6f\""
Sep 15 11:43:52.619296 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.619271    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l"]
Sep 15 11:43:52.662602 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.662562    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm"]
Sep 15 11:43:52.665821 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.665804    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm"
Sep 15 11:43:52.668895 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.668875    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-5a8d6f-dockercfg-dks7l\""
Sep 15 11:43:52.679372 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.679343    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm"]
Sep 15 11:43:52.729694 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.729662    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8e4f44-3327-49f4-b58a-e2d6d7653e6e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm\" (UID: \"1d8e4f44-3327-49f4-b58a-e2d6d7653e6e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm"
Sep 15 11:43:52.729885 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.729708    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86c579c-5c55-4e23-8a92-2fe23becf514-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l\" (UID: \"b86c579c-5c55-4e23-8a92-2fe23becf514\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l"
Sep 15 11:43:52.729885 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.729741    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gcw2\" (UniqueName: \"kubernetes.io/projected/b86c579c-5c55-4e23-8a92-2fe23becf514-kube-api-access-7gcw2\") pod \"payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l\" (UID: \"b86c579c-5c55-4e23-8a92-2fe23becf514\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l"
Sep 15 11:43:52.729885 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.729807    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2\" (UID: \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"
Sep 15 11:43:52.729885 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.729833    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxt66\" (UniqueName: \"kubernetes.io/projected/1d8e4f44-3327-49f4-b58a-e2d6d7653e6e-kube-api-access-mxt66\") pod \"payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm\" (UID: \"1d8e4f44-3327-49f4-b58a-e2d6d7653e6e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm"
Sep 15 11:43:52.732303 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.732271    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2\" (UID: \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"
Sep 15 11:43:52.830469 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.830427    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gcw2\" (UniqueName: \"kubernetes.io/projected/b86c579c-5c55-4e23-8a92-2fe23becf514-kube-api-access-7gcw2\") pod \"payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l\" (UID: \"b86c579c-5c55-4e23-8a92-2fe23becf514\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l"
Sep 15 11:43:52.830672 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.830495    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxt66\" (UniqueName: \"kubernetes.io/projected/1d8e4f44-3327-49f4-b58a-e2d6d7653e6e-kube-api-access-mxt66\") pod \"payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm\" (UID: \"1d8e4f44-3327-49f4-b58a-e2d6d7653e6e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm"
Sep 15 11:43:52.830672 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.830552    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8e4f44-3327-49f4-b58a-e2d6d7653e6e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm\" (UID: \"1d8e4f44-3327-49f4-b58a-e2d6d7653e6e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm"
Sep 15 11:43:52.830672 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.830598    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86c579c-5c55-4e23-8a92-2fe23becf514-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l\" (UID: \"b86c579c-5c55-4e23-8a92-2fe23becf514\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l"
Sep 15 11:43:52.831315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.831293    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86c579c-5c55-4e23-8a92-2fe23becf514-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l\" (UID: \"b86c579c-5c55-4e23-8a92-2fe23becf514\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l"
Sep 15 11:43:52.831315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.831296    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8e4f44-3327-49f4-b58a-e2d6d7653e6e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm\" (UID: \"1d8e4f44-3327-49f4-b58a-e2d6d7653e6e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm"
Sep 15 11:43:52.841044 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.841013    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gcw2\" (UniqueName: \"kubernetes.io/projected/b86c579c-5c55-4e23-8a92-2fe23becf514-kube-api-access-7gcw2\") pod \"payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l\" (UID: \"b86c579c-5c55-4e23-8a92-2fe23becf514\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l"
Sep 15 11:43:52.841166 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.841016    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxt66\" (UniqueName: \"kubernetes.io/projected/1d8e4f44-3327-49f4-b58a-e2d6d7653e6e-kube-api-access-mxt66\") pod \"payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm\" (UID: \"1d8e4f44-3327-49f4-b58a-e2d6d7653e6e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm"
Sep 15 11:43:52.901140 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.901021    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"
Sep 15 11:43:52.917118 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.917093    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l"
Sep 15 11:43:52.976123 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:52.975682    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm"
Sep 15 11:43:53.177280 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:53.177251    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"]
Sep 15 11:43:53.179704 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:43:53.179669    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5899f8_dd4e_4dfb_b3ea_d5d7811bef7b.slice/crio-9831291eafaeeba25c3c2043eda33f7b188946b01d63442f63959d65c6e2ca3a WatchSource:0}: Error finding container 9831291eafaeeba25c3c2043eda33f7b188946b01d63442f63959d65c6e2ca3a: Status 404 returned error can't find the container with id 9831291eafaeeba25c3c2043eda33f7b188946b01d63442f63959d65c6e2ca3a
Sep 15 11:43:53.181364 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:53.181339    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 11:43:53.201507 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:53.201415    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l"]
Sep 15 11:43:53.203506 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:43:53.203480    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86c579c_5c55_4e23_8a92_2fe23becf514.slice/crio-686b02ca4841edfef1ebdc90f175a2afc6a5e2c0e17aa7700f4c7120dc79b8c5 WatchSource:0}: Error finding container 686b02ca4841edfef1ebdc90f175a2afc6a5e2c0e17aa7700f4c7120dc79b8c5: Status 404 returned error can't find the container with id 686b02ca4841edfef1ebdc90f175a2afc6a5e2c0e17aa7700f4c7120dc79b8c5
Sep 15 11:43:53.257996 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:53.257922    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm"]
Sep 15 11:43:53.274183 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:43:53.274155    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8e4f44_3327_49f4_b58a_e2d6d7653e6e.slice/crio-e4061c400cddd71927679de5dc79b25a4d55930d1e37ba44b57507bb8f9077ae WatchSource:0}: Error finding container e4061c400cddd71927679de5dc79b25a4d55930d1e37ba44b57507bb8f9077ae: Status 404 returned error can't find the container with id e4061c400cddd71927679de5dc79b25a4d55930d1e37ba44b57507bb8f9077ae
Sep 15 11:43:53.862806 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:53.862765    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2" event={"ID":"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b","Type":"ContainerStarted","Data":"242556fe2fc3538a4f184112b43c19c48374c62c12c9e9cb7a9316d9eba75938"}
Sep 15 11:43:53.862996 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:53.862833    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2" event={"ID":"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b","Type":"ContainerStarted","Data":"9831291eafaeeba25c3c2043eda33f7b188946b01d63442f63959d65c6e2ca3a"}
Sep 15 11:43:53.862996 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:53.862940    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"
Sep 15 11:43:53.864456 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:53.864433    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm" event={"ID":"1d8e4f44-3327-49f4-b58a-e2d6d7653e6e","Type":"ContainerStarted","Data":"4abb53605609df859cf0d7cf547db4889e25e1436618a10d996e5dbd7d351325"}
Sep 15 11:43:53.864585 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:53.864481    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm"
Sep 15 11:43:53.864585 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:53.864503    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm" event={"ID":"1d8e4f44-3327-49f4-b58a-e2d6d7653e6e","Type":"ContainerStarted","Data":"e4061c400cddd71927679de5dc79b25a4d55930d1e37ba44b57507bb8f9077ae"}
Sep 15 11:43:53.878541 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:53.878511    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l"
Sep 15 11:43:53.878659 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:53.878554    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l" event={"ID":"b86c579c-5c55-4e23-8a92-2fe23becf514","Type":"ContainerStarted","Data":"a8df4aab47458606e0cc1753794766b4344d61ee64da71718de34fbc9febf7ed"}
Sep 15 11:43:53.878659 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:53.878574    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l" event={"ID":"b86c579c-5c55-4e23-8a92-2fe23becf514","Type":"ContainerStarted","Data":"686b02ca4841edfef1ebdc90f175a2afc6a5e2c0e17aa7700f4c7120dc79b8c5"}
Sep 15 11:43:53.881572 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:53.881529    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2" podStartSLOduration=2.429141595 podStartE2EDuration="2.881514829s" totalImagesPullingTime="452.373234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:43:53.880750785 +0000 UTC m=+1132.543245982" watchObservedRunningTime="2026-09-15 11:43:53.881514829 +0000 UTC m=+1132.544010010"
Sep 15 11:43:53.901428 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:53.901342    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm" podStartSLOduration=1.9013301550000001 podStartE2EDuration="1.901330155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:43:53.899819492 +0000 UTC m=+1132.562314706" watchObservedRunningTime="2026-09-15 11:43:53.901330155 +0000 UTC m=+1132.563825383"
Sep 15 11:43:53.920038 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:53.919994    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l" podStartSLOduration=1.9199796519999999 podStartE2EDuration="1.919979652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:43:53.918715508 +0000 UTC m=+1132.581210708" watchObservedRunningTime="2026-09-15 11:43:53.919979652 +0000 UTC m=+1132.582474833"
Sep 15 11:43:59.882301 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:43:59.882272    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"
Sep 15 11:44:04.878649 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:04.878620    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm"
Sep 15 11:44:04.879079 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:04.878694    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l"
Sep 15 11:44:20.654517 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:20.654473    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:44:27.814197 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:27.814114    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j"]
Sep 15 11:44:27.817777 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:27.817758    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j"
Sep 15 11:44:27.820032 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:27.820012    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-d63075\""
Sep 15 11:44:27.820222 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:27.820202    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-d63075-dockercfg-jzl99\""
Sep 15 11:44:27.826288 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:27.826266    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j"]
Sep 15 11:44:27.851697 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:27.851665    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad53e7e-8857-4115-a455-bee00e78eb6e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j\" (UID: \"8ad53e7e-8857-4115-a455-bee00e78eb6e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j"
Sep 15 11:44:27.851857 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:27.851791    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg7w4\" (UniqueName: \"kubernetes.io/projected/8ad53e7e-8857-4115-a455-bee00e78eb6e-kube-api-access-fg7w4\") pod \"payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j\" (UID: \"8ad53e7e-8857-4115-a455-bee00e78eb6e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j"
Sep 15 11:44:27.953016 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:27.952977    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg7w4\" (UniqueName: \"kubernetes.io/projected/8ad53e7e-8857-4115-a455-bee00e78eb6e-kube-api-access-fg7w4\") pod \"payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j\" (UID: \"8ad53e7e-8857-4115-a455-bee00e78eb6e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j"
Sep 15 11:44:27.953270 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:27.953096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad53e7e-8857-4115-a455-bee00e78eb6e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j\" (UID: \"8ad53e7e-8857-4115-a455-bee00e78eb6e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j"
Sep 15 11:44:27.953772 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:27.953753    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad53e7e-8857-4115-a455-bee00e78eb6e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j\" (UID: \"8ad53e7e-8857-4115-a455-bee00e78eb6e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j"
Sep 15 11:44:27.961655 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:27.961624    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg7w4\" (UniqueName: \"kubernetes.io/projected/8ad53e7e-8857-4115-a455-bee00e78eb6e-kube-api-access-fg7w4\") pod \"payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j\" (UID: \"8ad53e7e-8857-4115-a455-bee00e78eb6e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j"
Sep 15 11:44:28.129313 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:28.129275    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j"
Sep 15 11:44:28.406736 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:28.406707    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j"]
Sep 15 11:44:28.409325 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:44:28.409295    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad53e7e_8857_4115_a455_bee00e78eb6e.slice/crio-04e194062bb2e19af57c7c7bd28c7e71f7d479f850a9a9fddc5845d6df07331e WatchSource:0}: Error finding container 04e194062bb2e19af57c7c7bd28c7e71f7d479f850a9a9fddc5845d6df07331e: Status 404 returned error can't find the container with id 04e194062bb2e19af57c7c7bd28c7e71f7d479f850a9a9fddc5845d6df07331e
Sep 15 11:44:29.039202 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:29.039160    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j" event={"ID":"8ad53e7e-8857-4115-a455-bee00e78eb6e","Type":"ContainerStarted","Data":"348252064dae31af81102b06caff65af1be9d8e2fb54fd27af663a6a23f83e67"}
Sep 15 11:44:29.039699 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:29.039224    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j" event={"ID":"8ad53e7e-8857-4115-a455-bee00e78eb6e","Type":"ContainerStarted","Data":"04e194062bb2e19af57c7c7bd28c7e71f7d479f850a9a9fddc5845d6df07331e"}
Sep 15 11:44:29.039699 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:29.039252    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j"
Sep 15 11:44:29.057365 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:29.057306    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j" podStartSLOduration=2.05728869 podStartE2EDuration="2.05728869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:44:29.055105008 +0000 UTC m=+1167.717600206" watchObservedRunningTime="2026-09-15 11:44:29.05728869 +0000 UTC m=+1167.719783931"
Sep 15 11:44:40.046837 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.046799    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j"
Sep 15 11:44:40.534328 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.534291    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"]
Sep 15 11:44:40.537737 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.537715    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:40.540637 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.540612    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ed9962\"/\"noten-model-ed9962-kserve-self-signed-certs\""
Sep 15 11:44:40.540772 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.540651    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ed9962\"/\"default-dockercfg-xdx2b\""
Sep 15 11:44:40.540772 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.540720    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ed9962\"/\"kube-root-ca.crt\""
Sep 15 11:44:40.540933 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.540912    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ed9962\"/\"openshift-service-ca.crt\""
Sep 15 11:44:40.550079 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.550031    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"]
Sep 15 11:44:40.680458 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.680418    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605a9436-0413-4f88-a071-e06d5ecec31f-tls-certs\") pod \"noten-model-ed9962-kserve-58c4b48bdc-h5g56\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") " pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:40.680641 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.680473    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-model-cache\") pod \"noten-model-ed9962-kserve-58c4b48bdc-h5g56\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") " pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:40.680641 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.680503    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-dshm\") pod \"noten-model-ed9962-kserve-58c4b48bdc-h5g56\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") " pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:40.680641 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.680539    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-home\") pod \"noten-model-ed9962-kserve-58c4b48bdc-h5g56\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") " pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:40.680641 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.680563    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwcfk\" (UniqueName: \"kubernetes.io/projected/605a9436-0413-4f88-a071-e06d5ecec31f-kube-api-access-vwcfk\") pod \"noten-model-ed9962-kserve-58c4b48bdc-h5g56\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") " pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:40.680641 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.680583    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-tmp-dir\") pod \"noten-model-ed9962-kserve-58c4b48bdc-h5g56\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") " pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:40.781520 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.781483    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-dshm\") pod \"noten-model-ed9962-kserve-58c4b48bdc-h5g56\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") " pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:40.781520 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.781520    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-home\") pod \"noten-model-ed9962-kserve-58c4b48bdc-h5g56\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") " pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:40.781785 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.781548    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwcfk\" (UniqueName: \"kubernetes.io/projected/605a9436-0413-4f88-a071-e06d5ecec31f-kube-api-access-vwcfk\") pod \"noten-model-ed9962-kserve-58c4b48bdc-h5g56\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") " pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:40.781785 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.781683    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-tmp-dir\") pod \"noten-model-ed9962-kserve-58c4b48bdc-h5g56\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") " pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:40.781922 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.781823    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605a9436-0413-4f88-a071-e06d5ecec31f-tls-certs\") pod \"noten-model-ed9962-kserve-58c4b48bdc-h5g56\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") " pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:40.781922 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.781893    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-model-cache\") pod \"noten-model-ed9962-kserve-58c4b48bdc-h5g56\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") " pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:40.782025 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.781929    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-home\") pod \"noten-model-ed9962-kserve-58c4b48bdc-h5g56\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") " pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:40.782025 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.781971    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-tmp-dir\") pod \"noten-model-ed9962-kserve-58c4b48bdc-h5g56\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") " pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:40.782251 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.782232    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-model-cache\") pod \"noten-model-ed9962-kserve-58c4b48bdc-h5g56\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") " pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:40.783840 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.783821    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-dshm\") pod \"noten-model-ed9962-kserve-58c4b48bdc-h5g56\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") " pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:40.784137 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.784121    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605a9436-0413-4f88-a071-e06d5ecec31f-tls-certs\") pod \"noten-model-ed9962-kserve-58c4b48bdc-h5g56\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") " pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:40.789630 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.789573    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwcfk\" (UniqueName: \"kubernetes.io/projected/605a9436-0413-4f88-a071-e06d5ecec31f-kube-api-access-vwcfk\") pod \"noten-model-ed9962-kserve-58c4b48bdc-h5g56\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") " pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:40.849708 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:40.849663    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:41.083797 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:41.083760    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:44:41.127386 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:41.127355    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"]
Sep 15 11:44:41.130873 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:44:41.130842    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod605a9436_0413_4f88_a071_e06d5ecec31f.slice/crio-407a8776d5b7f728c1b72663469bc357e46f120ce65d3a3f73a0e0e3b8b24222 WatchSource:0}: Error finding container 407a8776d5b7f728c1b72663469bc357e46f120ce65d3a3f73a0e0e3b8b24222: Status 404 returned error can't find the container with id 407a8776d5b7f728c1b72663469bc357e46f120ce65d3a3f73a0e0e3b8b24222
Sep 15 11:44:42.110036 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:42.109994    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56" event={"ID":"605a9436-0413-4f88-a071-e06d5ecec31f","Type":"ContainerStarted","Data":"82e21883bd23812e6cb9da694857d6ca52b8a2db796784535d105121414a9d0f"}
Sep 15 11:44:42.110490 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:42.110073    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56" event={"ID":"605a9436-0413-4f88-a071-e06d5ecec31f","Type":"ContainerStarted","Data":"407a8776d5b7f728c1b72663469bc357e46f120ce65d3a3f73a0e0e3b8b24222"}
Sep 15 11:44:42.131368 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:42.131309    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56" podStartSLOduration=2.131292729 podStartE2EDuration="2.131292729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:44:42.127789741 +0000 UTC m=+1180.790284933" watchObservedRunningTime="2026-09-15 11:44:42.131292729 +0000 UTC m=+1180.793787910"
Sep 15 11:44:50.849921 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:50.849880    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:50.850443 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:50.849940    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:50.862572 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:50.862546    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:51.167534 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:51.167450    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:44:53.695439 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:53.695398    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7456-ef582e"]
Sep 15 11:44:53.699034 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:53.699014    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-ef582e"
Sep 15 11:44:53.701350 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:53.701155    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lfxxr\""
Sep 15 11:44:53.703696 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:53.703674    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7456-ef582e"]
Sep 15 11:44:53.806699 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:53.806661    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vttcp\" (UniqueName: \"kubernetes.io/projected/053a8ba7-117e-4de5-9266-4a1df03f8d70-kube-api-access-vttcp\") pod \"test-curl-7456-ef582e\" (UID: \"053a8ba7-117e-4de5-9266-4a1df03f8d70\") " pod="opendatahub/test-curl-7456-ef582e"
Sep 15 11:44:53.908191 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:53.908158    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vttcp\" (UniqueName: \"kubernetes.io/projected/053a8ba7-117e-4de5-9266-4a1df03f8d70-kube-api-access-vttcp\") pod \"test-curl-7456-ef582e\" (UID: \"053a8ba7-117e-4de5-9266-4a1df03f8d70\") " pod="opendatahub/test-curl-7456-ef582e"
Sep 15 11:44:53.916044 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:53.916009    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vttcp\" (UniqueName: \"kubernetes.io/projected/053a8ba7-117e-4de5-9266-4a1df03f8d70-kube-api-access-vttcp\") pod \"test-curl-7456-ef582e\" (UID: \"053a8ba7-117e-4de5-9266-4a1df03f8d70\") " pod="opendatahub/test-curl-7456-ef582e"
Sep 15 11:44:54.011820 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:54.011731    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-ef582e"
Sep 15 11:44:54.282843 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:54.282740    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7456-ef582e"]
Sep 15 11:44:54.285793 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:44:54.285767    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod053a8ba7_117e_4de5_9266_4a1df03f8d70.slice/crio-9ddd369c69a6b7e0b2f6645d1766436ee5fed30f6bc2753eb9e9a7ec01785e4f WatchSource:0}: Error finding container 9ddd369c69a6b7e0b2f6645d1766436ee5fed30f6bc2753eb9e9a7ec01785e4f: Status 404 returned error can't find the container with id 9ddd369c69a6b7e0b2f6645d1766436ee5fed30f6bc2753eb9e9a7ec01785e4f
Sep 15 11:44:55.177960 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:55.177922    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7456-ef582e" event={"ID":"053a8ba7-117e-4de5-9266-4a1df03f8d70","Type":"ContainerStarted","Data":"b81ece4de08e0be96c3cbbf65b66f06bc9c845dee66a99b706686fb2380e38d5"}
Sep 15 11:44:55.177960 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:55.177966    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7456-ef582e" event={"ID":"053a8ba7-117e-4de5-9266-4a1df03f8d70","Type":"ContainerStarted","Data":"9ddd369c69a6b7e0b2f6645d1766436ee5fed30f6bc2753eb9e9a7ec01785e4f"}
Sep 15 11:44:55.207911 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:55.207858    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7456-ef582e" podStartSLOduration=2.207841694 podStartE2EDuration="2.207841694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:44:55.20598773 +0000 UTC m=+1193.868482905" watchObservedRunningTime="2026-09-15 11:44:55.207841694 +0000 UTC m=+1193.870336874"
Sep 15 11:44:55.795616 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:55.795562    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7456-ef582e"]
Sep 15 11:44:55.800887 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:55.800852    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7456-ef582e"]
Sep 15 11:44:56.182093 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:44:56.182038    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7456-ef582e" container="test-curl-7456-ef582e"
Sep 15 11:44:57.187793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:57.187745    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7456-ef582e" podUID="053a8ba7-117e-4de5-9266-4a1df03f8d70" containerName="test-curl-7456-ef582e" containerID="cri-o://b81ece4de08e0be96c3cbbf65b66f06bc9c845dee66a99b706686fb2380e38d5" gracePeriod=2
Sep 15 11:44:59.334880 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:59.334852    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-ef582e"
Sep 15 11:44:59.464357 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:59.464275    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vttcp\" (UniqueName: \"kubernetes.io/projected/053a8ba7-117e-4de5-9266-4a1df03f8d70-kube-api-access-vttcp\") pod \"053a8ba7-117e-4de5-9266-4a1df03f8d70\" (UID: \"053a8ba7-117e-4de5-9266-4a1df03f8d70\") "
Sep 15 11:44:59.466340 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:59.466314    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053a8ba7-117e-4de5-9266-4a1df03f8d70-kube-api-access-vttcp" (OuterVolumeSpecName: "kube-api-access-vttcp") pod "053a8ba7-117e-4de5-9266-4a1df03f8d70" (UID: "053a8ba7-117e-4de5-9266-4a1df03f8d70"). InnerVolumeSpecName "kube-api-access-vttcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:44:59.565492 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:59.565442    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vttcp\" (UniqueName: \"kubernetes.io/projected/053a8ba7-117e-4de5-9266-4a1df03f8d70-kube-api-access-vttcp\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:44:59.874382 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:44:59.874346    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="053a8ba7-117e-4de5-9266-4a1df03f8d70" path="/var/lib/kubelet/pods/053a8ba7-117e-4de5-9266-4a1df03f8d70/volumes"
Sep 15 11:45:00.149574 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:00.149482    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-78522e-29824545-8mvtz"]
Sep 15 11:45:00.149949 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:00.149929    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="053a8ba7-117e-4de5-9266-4a1df03f8d70" containerName="test-curl-7456-ef582e"
Sep 15 11:45:00.150036 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:00.149951    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="053a8ba7-117e-4de5-9266-4a1df03f8d70" containerName="test-curl-7456-ef582e"
Sep 15 11:45:00.150036 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:00.150030    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="053a8ba7-117e-4de5-9266-4a1df03f8d70" containerName="test-curl-7456-ef582e"
Sep 15 11:45:00.152979 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:00.152961    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-78522e-29824545-8mvtz"
Sep 15 11:45:00.156942 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:00.156923    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rbxjk\""
Sep 15 11:45:00.162289 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:00.162273    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-78522e-29824545-8mvtz"
Sep 15 11:45:00.184173 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:00.184139    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-78522e-29824545-8mvtz"]
Sep 15 11:45:00.212533 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:00.212495    2562 generic.go:356] "Generic (PLEG): container finished" podID="053a8ba7-117e-4de5-9266-4a1df03f8d70" containerID="b81ece4de08e0be96c3cbbf65b66f06bc9c845dee66a99b706686fb2380e38d5" exitCode=137
Sep 15 11:45:00.212822 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:00.212806    2562 scope.go:117] "RemoveContainer" containerID="b81ece4de08e0be96c3cbbf65b66f06bc9c845dee66a99b706686fb2380e38d5"
Sep 15 11:45:00.214445 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:00.214254    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-ef582e"
Sep 15 11:45:00.224547 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:00.224526    2562 scope.go:117] "RemoveContainer" containerID="b81ece4de08e0be96c3cbbf65b66f06bc9c845dee66a99b706686fb2380e38d5"
Sep 15 11:45:00.225015 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:45:00.224997    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b81ece4de08e0be96c3cbbf65b66f06bc9c845dee66a99b706686fb2380e38d5\": container with ID starting with b81ece4de08e0be96c3cbbf65b66f06bc9c845dee66a99b706686fb2380e38d5 not found: ID does not exist" containerID="b81ece4de08e0be96c3cbbf65b66f06bc9c845dee66a99b706686fb2380e38d5"
Sep 15 11:45:00.225132 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:00.225075    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b81ece4de08e0be96c3cbbf65b66f06bc9c845dee66a99b706686fb2380e38d5"} err="failed to get container status \"b81ece4de08e0be96c3cbbf65b66f06bc9c845dee66a99b706686fb2380e38d5\": rpc error: code = NotFound desc = could not find container \"b81ece4de08e0be96c3cbbf65b66f06bc9c845dee66a99b706686fb2380e38d5\": container with ID starting with b81ece4de08e0be96c3cbbf65b66f06bc9c845dee66a99b706686fb2380e38d5 not found: ID does not exist"
Sep 15 11:45:00.451283 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:00.451254    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-78522e-29824545-8mvtz"]
Sep 15 11:45:00.453563 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:45:00.453534    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cbe476a_0127_44e7_a5c6_c320eb0b27b6.slice/crio-42896f7ceaaac3fab046ff35679bee11fc4d835aafe346d15334d21260c1577a WatchSource:0}: Error finding container 42896f7ceaaac3fab046ff35679bee11fc4d835aafe346d15334d21260c1577a: Status 404 returned error can't find the container with id 42896f7ceaaac3fab046ff35679bee11fc4d835aafe346d15334d21260c1577a
Sep 15 11:45:01.038609 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:01.038568    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7456-866444"]
Sep 15 11:45:01.043444 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:01.043420    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-866444"
Sep 15 11:45:01.045661 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:01.045639    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lfxxr\""
Sep 15 11:45:01.049183 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:01.049160    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7456-866444"]
Sep 15 11:45:01.181320 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:01.181283    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddcnc\" (UniqueName: \"kubernetes.io/projected/f12695ce-ef7e-4430-b160-e42fc1f9b111-kube-api-access-ddcnc\") pod \"test-curl-7456-866444\" (UID: \"f12695ce-ef7e-4430-b160-e42fc1f9b111\") " pod="opendatahub/test-curl-7456-866444"
Sep 15 11:45:01.219103 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:01.219049    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-78522e-29824545-8mvtz" event={"ID":"7cbe476a-0127-44e7-a5c6-c320eb0b27b6","Type":"ContainerStarted","Data":"42896f7ceaaac3fab046ff35679bee11fc4d835aafe346d15334d21260c1577a"}
Sep 15 11:45:01.282782 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:01.282736    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddcnc\" (UniqueName: \"kubernetes.io/projected/f12695ce-ef7e-4430-b160-e42fc1f9b111-kube-api-access-ddcnc\") pod \"test-curl-7456-866444\" (UID: \"f12695ce-ef7e-4430-b160-e42fc1f9b111\") " pod="opendatahub/test-curl-7456-866444"
Sep 15 11:45:01.291626 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:01.291552    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddcnc\" (UniqueName: \"kubernetes.io/projected/f12695ce-ef7e-4430-b160-e42fc1f9b111-kube-api-access-ddcnc\") pod \"test-curl-7456-866444\" (UID: \"f12695ce-ef7e-4430-b160-e42fc1f9b111\") " pod="opendatahub/test-curl-7456-866444"
Sep 15 11:45:01.354298 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:01.354262    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-866444"
Sep 15 11:45:01.626041 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:01.626015    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7456-866444"]
Sep 15 11:45:01.628245 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:45:01.628215    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf12695ce_ef7e_4430_b160_e42fc1f9b111.slice/crio-bb730211b27ae3501d04f634d1a986b046f11d8cb0511c4f99a9f854fc39291c WatchSource:0}: Error finding container bb730211b27ae3501d04f634d1a986b046f11d8cb0511c4f99a9f854fc39291c: Status 404 returned error can't find the container with id bb730211b27ae3501d04f634d1a986b046f11d8cb0511c4f99a9f854fc39291c
Sep 15 11:45:02.227874 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:02.227770    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7456-866444" event={"ID":"f12695ce-ef7e-4430-b160-e42fc1f9b111","Type":"ContainerStarted","Data":"5428c2da870a77a81156e8f862a8ddbda243f9c4c821bac0d4a20ec61b743dd1"}
Sep 15 11:45:02.227874 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:02.227828    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7456-866444" event={"ID":"f12695ce-ef7e-4430-b160-e42fc1f9b111","Type":"ContainerStarted","Data":"bb730211b27ae3501d04f634d1a986b046f11d8cb0511c4f99a9f854fc39291c"}
Sep 15 11:45:02.242811 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:02.242756    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7456-866444" podStartSLOduration=1.242739143 podStartE2EDuration="1.242739143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:02.241294356 +0000 UTC m=+1200.903789538" watchObservedRunningTime="2026-09-15 11:45:02.242739143 +0000 UTC m=+1200.905234323"
Sep 15 11:45:02.861073 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:02.861014    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7456-866444"]
Sep 15 11:45:02.865547 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:02.865516    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7456-866444"]
Sep 15 11:45:03.233606 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:45:03.233573    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7456-866444" container="test-curl-7456-866444"
Sep 15 11:45:04.239882 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:04.239831    2562 generic.go:356] "Generic (PLEG): container finished" podID="7cbe476a-0127-44e7-a5c6-c320eb0b27b6" containerID="6dd8bfb6c5937648468ad56ce036107147e456859aaef8388338c0bbc986191a" exitCode=0
Sep 15 11:45:04.240320 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:04.239912    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-78522e-29824545-8mvtz" event={"ID":"7cbe476a-0127-44e7-a5c6-c320eb0b27b6","Type":"ContainerDied","Data":"6dd8bfb6c5937648468ad56ce036107147e456859aaef8388338c0bbc986191a"}
Sep 15 11:45:04.240320 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:04.240096    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7456-866444" podUID="f12695ce-ef7e-4430-b160-e42fc1f9b111" containerName="test-curl-7456-866444" containerID="cri-o://5428c2da870a77a81156e8f862a8ddbda243f9c4c821bac0d4a20ec61b743dd1" gracePeriod=2
Sep 15 11:45:05.380876 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:05.380847    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-78522e-29824545-8mvtz"
Sep 15 11:45:06.250582 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:06.250557    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-78522e-29824545-8mvtz"
Sep 15 11:45:06.250582 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:06.250562    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-78522e-29824545-8mvtz" event={"ID":"7cbe476a-0127-44e7-a5c6-c320eb0b27b6","Type":"ContainerDied","Data":"42896f7ceaaac3fab046ff35679bee11fc4d835aafe346d15334d21260c1577a"}
Sep 15 11:45:06.250764 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:06.250609    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42896f7ceaaac3fab046ff35679bee11fc4d835aafe346d15334d21260c1577a"
Sep 15 11:45:06.413283 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:06.413250    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-866444"
Sep 15 11:45:06.433431 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:06.433400    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddcnc\" (UniqueName: \"kubernetes.io/projected/f12695ce-ef7e-4430-b160-e42fc1f9b111-kube-api-access-ddcnc\") pod \"f12695ce-ef7e-4430-b160-e42fc1f9b111\" (UID: \"f12695ce-ef7e-4430-b160-e42fc1f9b111\") "
Sep 15 11:45:06.435489 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:06.435468    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12695ce-ef7e-4430-b160-e42fc1f9b111-kube-api-access-ddcnc" (OuterVolumeSpecName: "kube-api-access-ddcnc") pod "f12695ce-ef7e-4430-b160-e42fc1f9b111" (UID: "f12695ce-ef7e-4430-b160-e42fc1f9b111"). InnerVolumeSpecName "kube-api-access-ddcnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:45:06.534226 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:06.534145    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddcnc\" (UniqueName: \"kubernetes.io/projected/f12695ce-ef7e-4430-b160-e42fc1f9b111-kube-api-access-ddcnc\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:45:07.256795 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:07.256755    2562 generic.go:356] "Generic (PLEG): container finished" podID="f12695ce-ef7e-4430-b160-e42fc1f9b111" containerID="5428c2da870a77a81156e8f862a8ddbda243f9c4c821bac0d4a20ec61b743dd1" exitCode=137
Sep 15 11:45:07.256969 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:07.256799    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7456-866444"
Sep 15 11:45:07.256969 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:07.256811    2562 scope.go:117] "RemoveContainer" containerID="5428c2da870a77a81156e8f862a8ddbda243f9c4c821bac0d4a20ec61b743dd1"
Sep 15 11:45:07.265565 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:07.265546    2562 scope.go:117] "RemoveContainer" containerID="5428c2da870a77a81156e8f862a8ddbda243f9c4c821bac0d4a20ec61b743dd1"
Sep 15 11:45:07.265834 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:45:07.265812    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5428c2da870a77a81156e8f862a8ddbda243f9c4c821bac0d4a20ec61b743dd1\": container with ID starting with 5428c2da870a77a81156e8f862a8ddbda243f9c4c821bac0d4a20ec61b743dd1 not found: ID does not exist" containerID="5428c2da870a77a81156e8f862a8ddbda243f9c4c821bac0d4a20ec61b743dd1"
Sep 15 11:45:07.265909 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:07.265846    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5428c2da870a77a81156e8f862a8ddbda243f9c4c821bac0d4a20ec61b743dd1"} err="failed to get container status \"5428c2da870a77a81156e8f862a8ddbda243f9c4c821bac0d4a20ec61b743dd1\": rpc error: code = NotFound desc = could not find container \"5428c2da870a77a81156e8f862a8ddbda243f9c4c821bac0d4a20ec61b743dd1\": container with ID starting with 5428c2da870a77a81156e8f862a8ddbda243f9c4c821bac0d4a20ec61b743dd1 not found: ID does not exist"
Sep 15 11:45:07.266935 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:45:07.266895    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7456-866444\" is forbidden: User \"system:node:ip-10-0-135-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-143.ec2.internal' and this object" podUID="f12695ce-ef7e-4430-b160-e42fc1f9b111" pod="opendatahub/test-curl-7456-866444"
Sep 15 11:45:07.878478 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:07.878428    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f12695ce-ef7e-4430-b160-e42fc1f9b111" path="/var/lib/kubelet/pods/f12695ce-ef7e-4430-b160-e42fc1f9b111/volumes"
Sep 15 11:45:08.775384 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:08.775351    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:45:13.374470 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:13.374436    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74"]
Sep 15 11:45:13.374864 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:13.374839    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cbe476a-0127-44e7-a5c6-c320eb0b27b6" containerName="cleanup"
Sep 15 11:45:13.374864 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:13.374854    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cbe476a-0127-44e7-a5c6-c320eb0b27b6" containerName="cleanup"
Sep 15 11:45:13.374964 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:13.374868    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f12695ce-ef7e-4430-b160-e42fc1f9b111" containerName="test-curl-7456-866444"
Sep 15 11:45:13.374964 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:13.374875    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f12695ce-ef7e-4430-b160-e42fc1f9b111" containerName="test-curl-7456-866444"
Sep 15 11:45:13.374964 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:13.374955    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f12695ce-ef7e-4430-b160-e42fc1f9b111" containerName="test-curl-7456-866444"
Sep 15 11:45:13.375081 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:13.374969    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cbe476a-0127-44e7-a5c6-c320eb0b27b6" containerName="cleanup"
Sep 15 11:45:13.379263 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:13.379245    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74"
Sep 15 11:45:13.381443 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:13.381419    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-baee2b93\""
Sep 15 11:45:13.381691 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:13.381673    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-baee2b93-dockercfg-tbxb7\""
Sep 15 11:45:13.388610 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:13.388585    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74"]
Sep 15 11:45:13.498992 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:13.498943    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g97dj\" (UniqueName: \"kubernetes.io/projected/ffcaffb2-0724-44a7-b998-d1b0b0dba00e-kube-api-access-g97dj\") pod \"payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74\" (UID: \"ffcaffb2-0724-44a7-b998-d1b0b0dba00e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74"
Sep 15 11:45:13.499200 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:13.499048    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffcaffb2-0724-44a7-b998-d1b0b0dba00e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74\" (UID: \"ffcaffb2-0724-44a7-b998-d1b0b0dba00e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74"
Sep 15 11:45:13.600254 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:13.600203    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g97dj\" (UniqueName: \"kubernetes.io/projected/ffcaffb2-0724-44a7-b998-d1b0b0dba00e-kube-api-access-g97dj\") pod \"payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74\" (UID: \"ffcaffb2-0724-44a7-b998-d1b0b0dba00e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74"
Sep 15 11:45:13.600438 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:13.600299    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffcaffb2-0724-44a7-b998-d1b0b0dba00e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74\" (UID: \"ffcaffb2-0724-44a7-b998-d1b0b0dba00e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74"
Sep 15 11:45:13.600978 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:13.600961    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffcaffb2-0724-44a7-b998-d1b0b0dba00e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74\" (UID: \"ffcaffb2-0724-44a7-b998-d1b0b0dba00e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74"
Sep 15 11:45:13.610552 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:13.610521    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g97dj\" (UniqueName: \"kubernetes.io/projected/ffcaffb2-0724-44a7-b998-d1b0b0dba00e-kube-api-access-g97dj\") pod \"payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74\" (UID: \"ffcaffb2-0724-44a7-b998-d1b0b0dba00e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74"
Sep 15 11:45:13.690131 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:13.690014    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74"
Sep 15 11:45:13.993002 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:13.992976    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74"]
Sep 15 11:45:13.994888 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:45:13.994860    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcaffb2_0724_44a7_b998_d1b0b0dba00e.slice/crio-0ecfbcf05567378d051c84efc13431fe2e107d53c64416e575e4aeac3da38a3c WatchSource:0}: Error finding container 0ecfbcf05567378d051c84efc13431fe2e107d53c64416e575e4aeac3da38a3c: Status 404 returned error can't find the container with id 0ecfbcf05567378d051c84efc13431fe2e107d53c64416e575e4aeac3da38a3c
Sep 15 11:45:14.292649 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:14.292542    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74" event={"ID":"ffcaffb2-0724-44a7-b998-d1b0b0dba00e","Type":"ContainerStarted","Data":"072d2f722630a252474e3018974e24de109c7e604a908fb5ca89ef359333221b"}
Sep 15 11:45:14.292649 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:14.292604    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74" event={"ID":"ffcaffb2-0724-44a7-b998-d1b0b0dba00e","Type":"ContainerStarted","Data":"0ecfbcf05567378d051c84efc13431fe2e107d53c64416e575e4aeac3da38a3c"}
Sep 15 11:45:14.292649 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:14.292637    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74"
Sep 15 11:45:14.310804 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:14.310756    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74" podStartSLOduration=1.310741307 podStartE2EDuration="1.310741307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:14.307801175 +0000 UTC m=+1212.970296359" watchObservedRunningTime="2026-09-15 11:45:14.310741307 +0000 UTC m=+1212.973236489"
Sep 15 11:45:16.997042 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:16.997008    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"]
Sep 15 11:45:25.299965 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:25.299931    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74"
Sep 15 11:45:32.016551 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.016494    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56" podUID="605a9436-0413-4f88-a071-e06d5ecec31f" containerName="main" containerID="cri-o://82e21883bd23812e6cb9da694857d6ca52b8a2db796784535d105121414a9d0f" gracePeriod=45
Sep 15 11:45:32.263299 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.263274    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:45:32.384122 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.384088    2562 generic.go:356] "Generic (PLEG): container finished" podID="605a9436-0413-4f88-a071-e06d5ecec31f" containerID="82e21883bd23812e6cb9da694857d6ca52b8a2db796784535d105121414a9d0f" exitCode=0
Sep 15 11:45:32.384305 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.384168    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"
Sep 15 11:45:32.384305 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.384209    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56" event={"ID":"605a9436-0413-4f88-a071-e06d5ecec31f","Type":"ContainerDied","Data":"82e21883bd23812e6cb9da694857d6ca52b8a2db796784535d105121414a9d0f"}
Sep 15 11:45:32.384305 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.384256    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56" event={"ID":"605a9436-0413-4f88-a071-e06d5ecec31f","Type":"ContainerDied","Data":"407a8776d5b7f728c1b72663469bc357e46f120ce65d3a3f73a0e0e3b8b24222"}
Sep 15 11:45:32.384305 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.384273    2562 scope.go:117] "RemoveContainer" containerID="82e21883bd23812e6cb9da694857d6ca52b8a2db796784535d105121414a9d0f"
Sep 15 11:45:32.386534 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.386514    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-tmp-dir\") pod \"605a9436-0413-4f88-a071-e06d5ecec31f\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") "
Sep 15 11:45:32.386657 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.386553    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605a9436-0413-4f88-a071-e06d5ecec31f-tls-certs\") pod \"605a9436-0413-4f88-a071-e06d5ecec31f\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") "
Sep 15 11:45:32.386657 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.386587    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-home\") pod \"605a9436-0413-4f88-a071-e06d5ecec31f\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") "
Sep 15 11:45:32.386745 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.386673    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwcfk\" (UniqueName: \"kubernetes.io/projected/605a9436-0413-4f88-a071-e06d5ecec31f-kube-api-access-vwcfk\") pod \"605a9436-0413-4f88-a071-e06d5ecec31f\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") "
Sep 15 11:45:32.386745 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.386725    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-dshm\") pod \"605a9436-0413-4f88-a071-e06d5ecec31f\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") "
Sep 15 11:45:32.386855 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.386811    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "605a9436-0413-4f88-a071-e06d5ecec31f" (UID: "605a9436-0413-4f88-a071-e06d5ecec31f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:45:32.386855 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.386852    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-model-cache\") pod \"605a9436-0413-4f88-a071-e06d5ecec31f\" (UID: \"605a9436-0413-4f88-a071-e06d5ecec31f\") "
Sep 15 11:45:32.386991 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.386964    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-home" (OuterVolumeSpecName: "home") pod "605a9436-0413-4f88-a071-e06d5ecec31f" (UID: "605a9436-0413-4f88-a071-e06d5ecec31f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:45:32.387282 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.387203    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-model-cache" (OuterVolumeSpecName: "model-cache") pod "605a9436-0413-4f88-a071-e06d5ecec31f" (UID: "605a9436-0413-4f88-a071-e06d5ecec31f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:45:32.387282 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.387246    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:45:32.387282 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.387265    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:45:32.388977 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.388938    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/605a9436-0413-4f88-a071-e06d5ecec31f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "605a9436-0413-4f88-a071-e06d5ecec31f" (UID: "605a9436-0413-4f88-a071-e06d5ecec31f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:45:32.389106 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.389043    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-dshm" (OuterVolumeSpecName: "dshm") pod "605a9436-0413-4f88-a071-e06d5ecec31f" (UID: "605a9436-0413-4f88-a071-e06d5ecec31f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:45:32.389106 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.389048    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605a9436-0413-4f88-a071-e06d5ecec31f-kube-api-access-vwcfk" (OuterVolumeSpecName: "kube-api-access-vwcfk") pod "605a9436-0413-4f88-a071-e06d5ecec31f" (UID: "605a9436-0413-4f88-a071-e06d5ecec31f"). InnerVolumeSpecName "kube-api-access-vwcfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:45:32.393116 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.393096    2562 scope.go:117] "RemoveContainer" containerID="82e21883bd23812e6cb9da694857d6ca52b8a2db796784535d105121414a9d0f"
Sep 15 11:45:32.393382 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:45:32.393365    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e21883bd23812e6cb9da694857d6ca52b8a2db796784535d105121414a9d0f\": container with ID starting with 82e21883bd23812e6cb9da694857d6ca52b8a2db796784535d105121414a9d0f not found: ID does not exist" containerID="82e21883bd23812e6cb9da694857d6ca52b8a2db796784535d105121414a9d0f"
Sep 15 11:45:32.393435 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.393391    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e21883bd23812e6cb9da694857d6ca52b8a2db796784535d105121414a9d0f"} err="failed to get container status \"82e21883bd23812e6cb9da694857d6ca52b8a2db796784535d105121414a9d0f\": rpc error: code = NotFound desc = could not find container \"82e21883bd23812e6cb9da694857d6ca52b8a2db796784535d105121414a9d0f\": container with ID starting with 82e21883bd23812e6cb9da694857d6ca52b8a2db796784535d105121414a9d0f not found: ID does not exist"
Sep 15 11:45:32.488417 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.488378    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwcfk\" (UniqueName: \"kubernetes.io/projected/605a9436-0413-4f88-a071-e06d5ecec31f-kube-api-access-vwcfk\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:45:32.488417 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.488409    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:45:32.488417 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.488426    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/605a9436-0413-4f88-a071-e06d5ecec31f-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:45:32.488651 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.488438    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/605a9436-0413-4f88-a071-e06d5ecec31f-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:45:32.705380 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.705348    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"]
Sep 15 11:45:32.709622 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:32.709598    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-ed9962/noten-model-ed9962-kserve-58c4b48bdc-h5g56"]
Sep 15 11:45:33.874974 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:33.874939    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="605a9436-0413-4f88-a071-e06d5ecec31f" path="/var/lib/kubelet/pods/605a9436-0413-4f88-a071-e06d5ecec31f/volumes"
Sep 15 11:45:45.057729 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:45:45.057694    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:46:07.458037 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:07.457993    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:46:23.453571 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.453520    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"]
Sep 15 11:46:23.454776 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.454706    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg" podUID="fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c" containerName="maas-api" containerID="cri-o://cc8d45d9e4592edb43f030302415bd59ddf408cfcf685cc0ffedea39b53bfcf2" gracePeriod=30
Sep 15 11:46:23.485723 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.485663    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-78522e-29824545-8mvtz"]
Sep 15 11:46:23.487650 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.487623    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-78522e-29824545-8mvtz"]
Sep 15 11:46:23.508144 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.508106    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5"]
Sep 15 11:46:23.508404 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.508370    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5" podUID="c3059f98-22e7-464b-8c09-a2c2d98d056f" containerName="payload-processing" containerID="cri-o://59e04cf708360c463d48845821dad2b1933b892aac0a2c3d141bbc6e51829ad6" gracePeriod=30
Sep 15 11:46:23.633305 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.633267    2562 generic.go:356] "Generic (PLEG): container finished" podID="fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c" containerID="cc8d45d9e4592edb43f030302415bd59ddf408cfcf685cc0ffedea39b53bfcf2" exitCode=0
Sep 15 11:46:23.633462 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.633352    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg" event={"ID":"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c","Type":"ContainerDied","Data":"cc8d45d9e4592edb43f030302415bd59ddf408cfcf685cc0ffedea39b53bfcf2"}
Sep 15 11:46:23.635198 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.635176    2562 generic.go:356] "Generic (PLEG): container finished" podID="c3059f98-22e7-464b-8c09-a2c2d98d056f" containerID="59e04cf708360c463d48845821dad2b1933b892aac0a2c3d141bbc6e51829ad6" exitCode=0
Sep 15 11:46:23.635317 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.635250    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5" event={"ID":"c3059f98-22e7-464b-8c09-a2c2d98d056f","Type":"ContainerDied","Data":"59e04cf708360c463d48845821dad2b1933b892aac0a2c3d141bbc6e51829ad6"}
Sep 15 11:46:23.760657 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.760630    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"
Sep 15 11:46:23.780802 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.778004    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smdv9\" (UniqueName: \"kubernetes.io/projected/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-kube-api-access-smdv9\") pod \"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c\" (UID: \"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c\") "
Sep 15 11:46:23.780802 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.778131    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-maas-api-tls\") pod \"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c\" (UID: \"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c\") "
Sep 15 11:46:23.780802 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.778242    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-metrics-certs\") pod \"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c\" (UID: \"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c\") "
Sep 15 11:46:23.780802 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.778324    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-openshift-service-ca\") pod \"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c\" (UID: \"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c\") "
Sep 15 11:46:23.780802 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.779282    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c" (UID: "fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:46:23.781623 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.781535    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-kube-api-access-smdv9" (OuterVolumeSpecName: "kube-api-access-smdv9") pod "fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c" (UID: "fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c"). InnerVolumeSpecName "kube-api-access-smdv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:46:23.781623 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.781543    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c" (UID: "fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:46:23.783408 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.783381    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c" (UID: "fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:46:23.791276 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.791257    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5"
Sep 15 11:46:23.874118 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.874085    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cbe476a-0127-44e7-a5c6-c320eb0b27b6" path="/var/lib/kubelet/pods/7cbe476a-0127-44e7-a5c6-c320eb0b27b6/volumes"
Sep 15 11:46:23.879267 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.879240    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsjnr\" (UniqueName: \"kubernetes.io/projected/c3059f98-22e7-464b-8c09-a2c2d98d056f-kube-api-access-qsjnr\") pod \"c3059f98-22e7-464b-8c09-a2c2d98d056f\" (UID: \"c3059f98-22e7-464b-8c09-a2c2d98d056f\") "
Sep 15 11:46:23.879403 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.879336    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3059f98-22e7-464b-8c09-a2c2d98d056f-plugins-config-volume\") pod \"c3059f98-22e7-464b-8c09-a2c2d98d056f\" (UID: \"c3059f98-22e7-464b-8c09-a2c2d98d056f\") "
Sep 15 11:46:23.879796 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.879613    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3059f98-22e7-464b-8c09-a2c2d98d056f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3059f98-22e7-464b-8c09-a2c2d98d056f" (UID: "c3059f98-22e7-464b-8c09-a2c2d98d056f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:46:23.879796 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.879714    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-maas-api-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:46:23.879796 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.879735    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3059f98-22e7-464b-8c09-a2c2d98d056f-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:46:23.879796 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.879750    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-metrics-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:46:23.879796 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.879763    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-openshift-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:46:23.879796 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.879777    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smdv9\" (UniqueName: \"kubernetes.io/projected/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c-kube-api-access-smdv9\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:46:23.881457 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.881430    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3059f98-22e7-464b-8c09-a2c2d98d056f-kube-api-access-qsjnr" (OuterVolumeSpecName: "kube-api-access-qsjnr") pod "c3059f98-22e7-464b-8c09-a2c2d98d056f" (UID: "c3059f98-22e7-464b-8c09-a2c2d98d056f"). InnerVolumeSpecName "kube-api-access-qsjnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:46:23.981156 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:23.981080    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsjnr\" (UniqueName: \"kubernetes.io/projected/c3059f98-22e7-464b-8c09-a2c2d98d056f-kube-api-access-qsjnr\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:46:24.643729 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:24.643699    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"
Sep 15 11:46:24.643729 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:24.643714    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg" event={"ID":"fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c","Type":"ContainerDied","Data":"94f6a91754077dddd4b019e93db844cb6252c2ff23cf4b4355bb3d08e9ff94fc"}
Sep 15 11:46:24.644294 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:24.643779    2562 scope.go:117] "RemoveContainer" containerID="cc8d45d9e4592edb43f030302415bd59ddf408cfcf685cc0ffedea39b53bfcf2"
Sep 15 11:46:24.645273 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:24.645252    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5" event={"ID":"c3059f98-22e7-464b-8c09-a2c2d98d056f","Type":"ContainerDied","Data":"f72ac2ae7ad47361c9b0c4694377f87ccb9294075477cb8ea0620815f7edc6ca"}
Sep 15 11:46:24.645370 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:24.645303    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5"
Sep 15 11:46:24.654823 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:24.654807    2562 scope.go:117] "RemoveContainer" containerID="59e04cf708360c463d48845821dad2b1933b892aac0a2c3d141bbc6e51829ad6"
Sep 15 11:46:24.664359 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:24.664337    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"]
Sep 15 11:46:24.668086 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:24.668036    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-78522e-66f659895d-drpkg"]
Sep 15 11:46:24.680784 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:24.680763    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5"]
Sep 15 11:46:24.690502 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:24.690482    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-78522e-f4b89754f-f8kx5"]
Sep 15 11:46:25.874213 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:25.874184    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3059f98-22e7-464b-8c09-a2c2d98d056f" path="/var/lib/kubelet/pods/c3059f98-22e7-464b-8c09-a2c2d98d056f/volumes"
Sep 15 11:46:25.874608 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:25.874594    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c" path="/var/lib/kubelet/pods/fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c/volumes"
Sep 15 11:46:29.838529 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.838489    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"]
Sep 15 11:46:29.838913 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.838871    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c" containerName="maas-api"
Sep 15 11:46:29.838913 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.838883    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c" containerName="maas-api"
Sep 15 11:46:29.838913 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.838894    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3059f98-22e7-464b-8c09-a2c2d98d056f" containerName="payload-processing"
Sep 15 11:46:29.838913 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.838902    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3059f98-22e7-464b-8c09-a2c2d98d056f" containerName="payload-processing"
Sep 15 11:46:29.839083 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.838918    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="605a9436-0413-4f88-a071-e06d5ecec31f" containerName="main"
Sep 15 11:46:29.839083 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.838923    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="605a9436-0413-4f88-a071-e06d5ecec31f" containerName="main"
Sep 15 11:46:29.839083 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.838988    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3059f98-22e7-464b-8c09-a2c2d98d056f" containerName="payload-processing"
Sep 15 11:46:29.839083 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.838998    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd3b74e3-6049-4f0a-a7eb-6eeddd98ea9c" containerName="maas-api"
Sep 15 11:46:29.839083 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.839005    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="605a9436-0413-4f88-a071-e06d5ecec31f" containerName="main"
Sep 15 11:46:29.843407 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.843388    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:29.846346 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.846321    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-baee2b93\"/\"kube-root-ca.crt\""
Sep 15 11:46:29.846480 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.846326    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-baee2b93\"/\"openshift-service-ca.crt\""
Sep 15 11:46:29.846480 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.846466    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-baee2b93\"/\"e2e-lifecycle-model-baee2b93-kserve-self-signed-certs\""
Sep 15 11:46:29.846587 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.846483    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-baee2b93\"/\"default-dockercfg-gldh6\""
Sep 15 11:46:29.851681 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.851657    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"]
Sep 15 11:46:29.939396 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.939362    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tq2w\" (UniqueName: \"kubernetes.io/projected/0885f161-820e-474a-af09-97634e62d961-kube-api-access-8tq2w\") pod \"e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") " pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:29.939561 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.939406    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-tmp-dir\") pod \"e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") " pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:29.939561 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.939502    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-model-cache\") pod \"e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") " pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:29.939730 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.939705    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-home\") pod \"e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") " pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:29.939799 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.939785    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-dshm\") pod \"e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") " pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:29.939852 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:29.939813    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0885f161-820e-474a-af09-97634e62d961-tls-certs\") pod \"e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") " pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:30.040840 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:30.040804    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tq2w\" (UniqueName: \"kubernetes.io/projected/0885f161-820e-474a-af09-97634e62d961-kube-api-access-8tq2w\") pod \"e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") " pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:30.040840 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:30.040847    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-tmp-dir\") pod \"e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") " pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:30.041154 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:30.040883    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-model-cache\") pod \"e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") " pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:30.041154 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:30.040922    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-home\") pod \"e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") " pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:30.041154 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:30.040956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-dshm\") pod \"e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") " pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:30.041154 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:30.040981    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0885f161-820e-474a-af09-97634e62d961-tls-certs\") pod \"e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") " pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:30.041341 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:30.041308    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-tmp-dir\") pod \"e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") " pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:30.041379 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:30.041361    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-home\") pod \"e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") " pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:30.041465 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:30.041438    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-model-cache\") pod \"e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") " pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:30.043355 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:30.043320    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-dshm\") pod \"e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") " pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:30.043636 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:30.043616    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0885f161-820e-474a-af09-97634e62d961-tls-certs\") pod \"e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") " pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:30.050997 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:30.050973    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tq2w\" (UniqueName: \"kubernetes.io/projected/0885f161-820e-474a-af09-97634e62d961-kube-api-access-8tq2w\") pod \"e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") " pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:30.154367 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:30.154281    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:30.426350 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:30.426274    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"]
Sep 15 11:46:30.429694 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:46:30.429665    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0885f161_820e_474a_af09_97634e62d961.slice/crio-15c50ee89c850b99459a6ffc05b11194d6f7cdb3e844d8ced810541917f7b8ec WatchSource:0}: Error finding container 15c50ee89c850b99459a6ffc05b11194d6f7cdb3e844d8ced810541917f7b8ec: Status 404 returned error can't find the container with id 15c50ee89c850b99459a6ffc05b11194d6f7cdb3e844d8ced810541917f7b8ec
Sep 15 11:46:30.677822 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:30.677725    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6" event={"ID":"0885f161-820e-474a-af09-97634e62d961","Type":"ContainerStarted","Data":"8ca2702f77a5dd9f1f9aaa8f33bf1598fb4bb131fc8824b3be21b58269d5895e"}
Sep 15 11:46:30.677822 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:30.677773    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6" event={"ID":"0885f161-820e-474a-af09-97634e62d961","Type":"ContainerStarted","Data":"15c50ee89c850b99459a6ffc05b11194d6f7cdb3e844d8ced810541917f7b8ec"}
Sep 15 11:46:30.694860 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:30.694806    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6" podStartSLOduration=1.694790316 podStartE2EDuration="1.694790316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:46:30.694113278 +0000 UTC m=+1289.356608469" watchObservedRunningTime="2026-09-15 11:46:30.694790316 +0000 UTC m=+1289.357285520"
Sep 15 11:46:38.760743 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:38.760704    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:46:40.155397 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:40.155360    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:40.155790 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:40.155621    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:40.168403 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:40.168380    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:40.737853 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:40.737815    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:46:43.538388 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:43.538350    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5"]
Sep 15 11:46:43.541894 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:43.541876    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5"
Sep 15 11:46:43.544163 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:43.544136    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-7efa8b-dockercfg-jssv6\""
Sep 15 11:46:43.544277 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:43.544190    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-7efa8b\""
Sep 15 11:46:43.559482 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:43.559453    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5"]
Sep 15 11:46:43.671837 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:43.671803    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7e1eca0-7a52-476e-9f5f-5ee74dcd7113-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5\" (UID: \"b7e1eca0-7a52-476e-9f5f-5ee74dcd7113\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5"
Sep 15 11:46:43.672043 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:43.671887    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqbsp\" (UniqueName: \"kubernetes.io/projected/b7e1eca0-7a52-476e-9f5f-5ee74dcd7113-kube-api-access-mqbsp\") pod \"payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5\" (UID: \"b7e1eca0-7a52-476e-9f5f-5ee74dcd7113\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5"
Sep 15 11:46:43.773119 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:43.773048    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqbsp\" (UniqueName: \"kubernetes.io/projected/b7e1eca0-7a52-476e-9f5f-5ee74dcd7113-kube-api-access-mqbsp\") pod \"payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5\" (UID: \"b7e1eca0-7a52-476e-9f5f-5ee74dcd7113\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5"
Sep 15 11:46:43.773305 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:43.773178    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7e1eca0-7a52-476e-9f5f-5ee74dcd7113-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5\" (UID: \"b7e1eca0-7a52-476e-9f5f-5ee74dcd7113\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5"
Sep 15 11:46:43.773883 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:43.773862    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7e1eca0-7a52-476e-9f5f-5ee74dcd7113-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5\" (UID: \"b7e1eca0-7a52-476e-9f5f-5ee74dcd7113\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5"
Sep 15 11:46:43.780890 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:43.780861    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqbsp\" (UniqueName: \"kubernetes.io/projected/b7e1eca0-7a52-476e-9f5f-5ee74dcd7113-kube-api-access-mqbsp\") pod \"payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5\" (UID: \"b7e1eca0-7a52-476e-9f5f-5ee74dcd7113\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5"
Sep 15 11:46:43.858961 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:43.858920    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5"
Sep 15 11:46:44.132317 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:44.132286    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5"]
Sep 15 11:46:44.135022 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:46:44.134980    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e1eca0_7a52_476e_9f5f_5ee74dcd7113.slice/crio-647b597bb22bc3860a4e709141b6f5f9524db56e3361fc93d5e73bdb778663c5 WatchSource:0}: Error finding container 647b597bb22bc3860a4e709141b6f5f9524db56e3361fc93d5e73bdb778663c5: Status 404 returned error can't find the container with id 647b597bb22bc3860a4e709141b6f5f9524db56e3361fc93d5e73bdb778663c5
Sep 15 11:46:44.751812 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:44.751774    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5" event={"ID":"b7e1eca0-7a52-476e-9f5f-5ee74dcd7113","Type":"ContainerStarted","Data":"b64e4b19e817f6f30dbd4faa013f9907be9cce07d98b7c7e9d00af9339ae6bda"}
Sep 15 11:46:44.752222 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:44.751838    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5" event={"ID":"b7e1eca0-7a52-476e-9f5f-5ee74dcd7113","Type":"ContainerStarted","Data":"647b597bb22bc3860a4e709141b6f5f9524db56e3361fc93d5e73bdb778663c5"}
Sep 15 11:46:44.752222 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:44.751869    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5"
Sep 15 11:46:44.768395 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:44.768304    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5" podStartSLOduration=1.768286538 podStartE2EDuration="1.768286538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:46:44.767460766 +0000 UTC m=+1303.429955947" watchObservedRunningTime="2026-09-15 11:46:44.768286538 +0000 UTC m=+1303.430781718"
Sep 15 11:46:55.759803 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:46:55.759765    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5"
Sep 15 11:47:12.255730 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:12.255692    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:47:18.809174 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:18.809139    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp"]
Sep 15 11:47:18.814005 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:18.813985    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp"
Sep 15 11:47:18.817200 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:18.817169    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-2cc790\""
Sep 15 11:47:18.817347 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:18.817181    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-2cc790-dockercfg-mknn5\""
Sep 15 11:47:18.819270 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:18.819250    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6"]
Sep 15 11:47:18.822639 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:18.822619    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6"
Sep 15 11:47:18.830981 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:18.830947    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp"]
Sep 15 11:47:18.857992 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:18.857958    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6"]
Sep 15 11:47:18.895440 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:18.895408    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6f4f0c-ebed-466b-b6af-19d74818d559-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6\" (UID: \"fa6f4f0c-ebed-466b-b6af-19d74818d559\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6"
Sep 15 11:47:18.895440 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:18.895441    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9x2h\" (UniqueName: \"kubernetes.io/projected/fa6f4f0c-ebed-466b-b6af-19d74818d559-kube-api-access-j9x2h\") pod \"payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6\" (UID: \"fa6f4f0c-ebed-466b-b6af-19d74818d559\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6"
Sep 15 11:47:18.895654 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:18.895463    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8080f8e1-0e28-4fef-a1da-9eedd98b5fd0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp\" (UID: \"8080f8e1-0e28-4fef-a1da-9eedd98b5fd0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp"
Sep 15 11:47:18.895654 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:18.895578    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9cwc\" (UniqueName: \"kubernetes.io/projected/8080f8e1-0e28-4fef-a1da-9eedd98b5fd0-kube-api-access-v9cwc\") pod \"payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp\" (UID: \"8080f8e1-0e28-4fef-a1da-9eedd98b5fd0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp"
Sep 15 11:47:18.996214 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:18.996176    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6f4f0c-ebed-466b-b6af-19d74818d559-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6\" (UID: \"fa6f4f0c-ebed-466b-b6af-19d74818d559\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6"
Sep 15 11:47:18.996214 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:18.996214    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9x2h\" (UniqueName: \"kubernetes.io/projected/fa6f4f0c-ebed-466b-b6af-19d74818d559-kube-api-access-j9x2h\") pod \"payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6\" (UID: \"fa6f4f0c-ebed-466b-b6af-19d74818d559\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6"
Sep 15 11:47:18.996429 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:18.996242    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8080f8e1-0e28-4fef-a1da-9eedd98b5fd0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp\" (UID: \"8080f8e1-0e28-4fef-a1da-9eedd98b5fd0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp"
Sep 15 11:47:18.996429 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:18.996334    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9cwc\" (UniqueName: \"kubernetes.io/projected/8080f8e1-0e28-4fef-a1da-9eedd98b5fd0-kube-api-access-v9cwc\") pod \"payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp\" (UID: \"8080f8e1-0e28-4fef-a1da-9eedd98b5fd0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp"
Sep 15 11:47:18.996917 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:18.996895    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6f4f0c-ebed-466b-b6af-19d74818d559-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6\" (UID: \"fa6f4f0c-ebed-466b-b6af-19d74818d559\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6"
Sep 15 11:47:18.997001 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:18.996982    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8080f8e1-0e28-4fef-a1da-9eedd98b5fd0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp\" (UID: \"8080f8e1-0e28-4fef-a1da-9eedd98b5fd0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp"
Sep 15 11:47:19.004984 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:19.004948    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9cwc\" (UniqueName: \"kubernetes.io/projected/8080f8e1-0e28-4fef-a1da-9eedd98b5fd0-kube-api-access-v9cwc\") pod \"payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp\" (UID: \"8080f8e1-0e28-4fef-a1da-9eedd98b5fd0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp"
Sep 15 11:47:19.005153 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:19.005036    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9x2h\" (UniqueName: \"kubernetes.io/projected/fa6f4f0c-ebed-466b-b6af-19d74818d559-kube-api-access-j9x2h\") pod \"payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6\" (UID: \"fa6f4f0c-ebed-466b-b6af-19d74818d559\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6"
Sep 15 11:47:19.124922 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:19.124870    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp"
Sep 15 11:47:19.133008 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:19.132989    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6"
Sep 15 11:47:19.402998 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:19.402967    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp"]
Sep 15 11:47:19.404995 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:47:19.404971    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8080f8e1_0e28_4fef_a1da_9eedd98b5fd0.slice/crio-237294044bca875a1f86457f9fbaef899e6e04d9462c274d0b992150475ec9c5 WatchSource:0}: Error finding container 237294044bca875a1f86457f9fbaef899e6e04d9462c274d0b992150475ec9c5: Status 404 returned error can't find the container with id 237294044bca875a1f86457f9fbaef899e6e04d9462c274d0b992150475ec9c5
Sep 15 11:47:19.411584 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:19.411560    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6"]
Sep 15 11:47:19.413460 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:47:19.413441    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6f4f0c_ebed_466b_b6af_19d74818d559.slice/crio-17d0bf311a53959280a3637f346a8e7b468bf866b816acf70aea4e725d387335 WatchSource:0}: Error finding container 17d0bf311a53959280a3637f346a8e7b468bf866b816acf70aea4e725d387335: Status 404 returned error can't find the container with id 17d0bf311a53959280a3637f346a8e7b468bf866b816acf70aea4e725d387335
Sep 15 11:47:19.939527 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:19.939488    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6" event={"ID":"fa6f4f0c-ebed-466b-b6af-19d74818d559","Type":"ContainerStarted","Data":"72d700bd74fe1821a1fd7cc9ef1e775c23dd17ef7b422d0bd4e465b50c1d791c"}
Sep 15 11:47:19.940039 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:19.939557    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6"
Sep 15 11:47:19.940039 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:19.939583    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6" event={"ID":"fa6f4f0c-ebed-466b-b6af-19d74818d559","Type":"ContainerStarted","Data":"17d0bf311a53959280a3637f346a8e7b468bf866b816acf70aea4e725d387335"}
Sep 15 11:47:19.941281 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:19.941252    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp" event={"ID":"8080f8e1-0e28-4fef-a1da-9eedd98b5fd0","Type":"ContainerStarted","Data":"6bc35efc27b4902474fb9fd88b8310e567f8398d1b4e92f59dc85a7e09109d2b"}
Sep 15 11:47:19.941399 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:19.941298    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp" event={"ID":"8080f8e1-0e28-4fef-a1da-9eedd98b5fd0","Type":"ContainerStarted","Data":"237294044bca875a1f86457f9fbaef899e6e04d9462c274d0b992150475ec9c5"}
Sep 15 11:47:19.941399 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:19.941332    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp"
Sep 15 11:47:19.956392 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:19.956347    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6" podStartSLOduration=1.956330554 podStartE2EDuration="1.956330554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:47:19.954732608 +0000 UTC m=+1338.617227810" watchObservedRunningTime="2026-09-15 11:47:19.956330554 +0000 UTC m=+1338.618825786"
Sep 15 11:47:19.970196 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:19.970148    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp" podStartSLOduration=1.970122513 podStartE2EDuration="1.970122513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:47:19.96980621 +0000 UTC m=+1338.632301415" watchObservedRunningTime="2026-09-15 11:47:19.970122513 +0000 UTC m=+1338.632617695"
Sep 15 11:47:30.948534 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:30.948456    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp"
Sep 15 11:47:30.948534 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:47:30.948525    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6"
Sep 15 11:48:21.157669 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:21.157628    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:48:45.834006 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:45.833969    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74"]
Sep 15 11:48:45.834563 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:45.834328    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74" podUID="ffcaffb2-0724-44a7-b998-d1b0b0dba00e" containerName="payload-pre-processing" containerID="cri-o://072d2f722630a252474e3018974e24de109c7e604a908fb5ca89ef359333221b" gracePeriod=30
Sep 15 11:48:46.087745 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:46.087692    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74"
Sep 15 11:48:46.201384 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:46.201350    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffcaffb2-0724-44a7-b998-d1b0b0dba00e-plugins-config-volume\") pod \"ffcaffb2-0724-44a7-b998-d1b0b0dba00e\" (UID: \"ffcaffb2-0724-44a7-b998-d1b0b0dba00e\") "
Sep 15 11:48:46.201574 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:46.201392    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g97dj\" (UniqueName: \"kubernetes.io/projected/ffcaffb2-0724-44a7-b998-d1b0b0dba00e-kube-api-access-g97dj\") pod \"ffcaffb2-0724-44a7-b998-d1b0b0dba00e\" (UID: \"ffcaffb2-0724-44a7-b998-d1b0b0dba00e\") "
Sep 15 11:48:46.201761 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:46.201736    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffcaffb2-0724-44a7-b998-d1b0b0dba00e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ffcaffb2-0724-44a7-b998-d1b0b0dba00e" (UID: "ffcaffb2-0724-44a7-b998-d1b0b0dba00e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:48:46.203424 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:46.203400    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcaffb2-0724-44a7-b998-d1b0b0dba00e-kube-api-access-g97dj" (OuterVolumeSpecName: "kube-api-access-g97dj") pod "ffcaffb2-0724-44a7-b998-d1b0b0dba00e" (UID: "ffcaffb2-0724-44a7-b998-d1b0b0dba00e"). InnerVolumeSpecName "kube-api-access-g97dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:48:46.302232 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:46.302203    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffcaffb2-0724-44a7-b998-d1b0b0dba00e-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:48:46.302232 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:46.302229    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g97dj\" (UniqueName: \"kubernetes.io/projected/ffcaffb2-0724-44a7-b998-d1b0b0dba00e-kube-api-access-g97dj\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:48:46.367107 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:46.367018    2562 generic.go:356] "Generic (PLEG): container finished" podID="ffcaffb2-0724-44a7-b998-d1b0b0dba00e" containerID="072d2f722630a252474e3018974e24de109c7e604a908fb5ca89ef359333221b" exitCode=0
Sep 15 11:48:46.367107 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:46.367072    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74"
Sep 15 11:48:46.367107 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:46.367084    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74" event={"ID":"ffcaffb2-0724-44a7-b998-d1b0b0dba00e","Type":"ContainerDied","Data":"072d2f722630a252474e3018974e24de109c7e604a908fb5ca89ef359333221b"}
Sep 15 11:48:46.367315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:46.367128    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74" event={"ID":"ffcaffb2-0724-44a7-b998-d1b0b0dba00e","Type":"ContainerDied","Data":"0ecfbcf05567378d051c84efc13431fe2e107d53c64416e575e4aeac3da38a3c"}
Sep 15 11:48:46.367315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:46.367149    2562 scope.go:117] "RemoveContainer" containerID="072d2f722630a252474e3018974e24de109c7e604a908fb5ca89ef359333221b"
Sep 15 11:48:46.378681 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:46.378655    2562 scope.go:117] "RemoveContainer" containerID="072d2f722630a252474e3018974e24de109c7e604a908fb5ca89ef359333221b"
Sep 15 11:48:46.379215 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:48:46.379191    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"072d2f722630a252474e3018974e24de109c7e604a908fb5ca89ef359333221b\": container with ID starting with 072d2f722630a252474e3018974e24de109c7e604a908fb5ca89ef359333221b not found: ID does not exist" containerID="072d2f722630a252474e3018974e24de109c7e604a908fb5ca89ef359333221b"
Sep 15 11:48:46.379309 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:46.379223    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"072d2f722630a252474e3018974e24de109c7e604a908fb5ca89ef359333221b"} err="failed to get container status \"072d2f722630a252474e3018974e24de109c7e604a908fb5ca89ef359333221b\": rpc error: code = NotFound desc = could not find container \"072d2f722630a252474e3018974e24de109c7e604a908fb5ca89ef359333221b\": container with ID starting with 072d2f722630a252474e3018974e24de109c7e604a908fb5ca89ef359333221b not found: ID does not exist"
Sep 15 11:48:46.391767 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:46.391745    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74"]
Sep 15 11:48:46.396066 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:46.396036    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-baee2b93-7bd7db4886-rth74"]
Sep 15 11:48:47.874275 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:47.874243    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffcaffb2-0724-44a7-b998-d1b0b0dba00e" path="/var/lib/kubelet/pods/ffcaffb2-0724-44a7-b998-d1b0b0dba00e/volumes"
Sep 15 11:48:51.561138 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:51.561102    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:48:54.141651 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:48:54.141570    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"]
Sep 15 11:49:09.160875 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.160815    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6" podUID="0885f161-820e-474a-af09-97634e62d961" containerName="main" containerID="cri-o://8ca2702f77a5dd9f1f9aaa8f33bf1598fb4bb131fc8824b3be21b58269d5895e" gracePeriod=45
Sep 15 11:49:09.419584 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.419531    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:49:09.478632 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.478595    2562 generic.go:356] "Generic (PLEG): container finished" podID="0885f161-820e-474a-af09-97634e62d961" containerID="8ca2702f77a5dd9f1f9aaa8f33bf1598fb4bb131fc8824b3be21b58269d5895e" exitCode=0
Sep 15 11:49:09.478632 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.478632    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"
Sep 15 11:49:09.478835 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.478634    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6" event={"ID":"0885f161-820e-474a-af09-97634e62d961","Type":"ContainerDied","Data":"8ca2702f77a5dd9f1f9aaa8f33bf1598fb4bb131fc8824b3be21b58269d5895e"}
Sep 15 11:49:09.478835 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.478680    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6" event={"ID":"0885f161-820e-474a-af09-97634e62d961","Type":"ContainerDied","Data":"15c50ee89c850b99459a6ffc05b11194d6f7cdb3e844d8ced810541917f7b8ec"}
Sep 15 11:49:09.478835 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.478700    2562 scope.go:117] "RemoveContainer" containerID="8ca2702f77a5dd9f1f9aaa8f33bf1598fb4bb131fc8824b3be21b58269d5895e"
Sep 15 11:49:09.487505 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.487486    2562 scope.go:117] "RemoveContainer" containerID="8ca2702f77a5dd9f1f9aaa8f33bf1598fb4bb131fc8824b3be21b58269d5895e"
Sep 15 11:49:09.487800 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:49:09.487782    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ca2702f77a5dd9f1f9aaa8f33bf1598fb4bb131fc8824b3be21b58269d5895e\": container with ID starting with 8ca2702f77a5dd9f1f9aaa8f33bf1598fb4bb131fc8824b3be21b58269d5895e not found: ID does not exist" containerID="8ca2702f77a5dd9f1f9aaa8f33bf1598fb4bb131fc8824b3be21b58269d5895e"
Sep 15 11:49:09.487874 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.487814    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ca2702f77a5dd9f1f9aaa8f33bf1598fb4bb131fc8824b3be21b58269d5895e"} err="failed to get container status \"8ca2702f77a5dd9f1f9aaa8f33bf1598fb4bb131fc8824b3be21b58269d5895e\": rpc error: code = NotFound desc = could not find container \"8ca2702f77a5dd9f1f9aaa8f33bf1598fb4bb131fc8824b3be21b58269d5895e\": container with ID starting with 8ca2702f77a5dd9f1f9aaa8f33bf1598fb4bb131fc8824b3be21b58269d5895e not found: ID does not exist"
Sep 15 11:49:09.503130 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.503103    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0885f161-820e-474a-af09-97634e62d961-tls-certs\") pod \"0885f161-820e-474a-af09-97634e62d961\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") "
Sep 15 11:49:09.503248 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.503138    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-tmp-dir\") pod \"0885f161-820e-474a-af09-97634e62d961\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") "
Sep 15 11:49:09.503315 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.503265    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-model-cache\") pod \"0885f161-820e-474a-af09-97634e62d961\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") "
Sep 15 11:49:09.503385 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.503319    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-home\") pod \"0885f161-820e-474a-af09-97634e62d961\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") "
Sep 15 11:49:09.503385 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.503366    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-dshm\") pod \"0885f161-820e-474a-af09-97634e62d961\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") "
Sep 15 11:49:09.503491 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.503413    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tq2w\" (UniqueName: \"kubernetes.io/projected/0885f161-820e-474a-af09-97634e62d961-kube-api-access-8tq2w\") pod \"0885f161-820e-474a-af09-97634e62d961\" (UID: \"0885f161-820e-474a-af09-97634e62d961\") "
Sep 15 11:49:09.503491 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.503412    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0885f161-820e-474a-af09-97634e62d961" (UID: "0885f161-820e-474a-af09-97634e62d961"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:49:09.503603 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.503529    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-home" (OuterVolumeSpecName: "home") pod "0885f161-820e-474a-af09-97634e62d961" (UID: "0885f161-820e-474a-af09-97634e62d961"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:49:09.503603 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.503540    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-model-cache" (OuterVolumeSpecName: "model-cache") pod "0885f161-820e-474a-af09-97634e62d961" (UID: "0885f161-820e-474a-af09-97634e62d961"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:49:09.503750 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.503732    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:49:09.503811 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.503755    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:49:09.503811 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.503769    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:49:09.505640 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.505617    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0885f161-820e-474a-af09-97634e62d961-kube-api-access-8tq2w" (OuterVolumeSpecName: "kube-api-access-8tq2w") pod "0885f161-820e-474a-af09-97634e62d961" (UID: "0885f161-820e-474a-af09-97634e62d961"). InnerVolumeSpecName "kube-api-access-8tq2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:49:09.505770 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.505751    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-dshm" (OuterVolumeSpecName: "dshm") pod "0885f161-820e-474a-af09-97634e62d961" (UID: "0885f161-820e-474a-af09-97634e62d961"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:49:09.505770 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.505748    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0885f161-820e-474a-af09-97634e62d961-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0885f161-820e-474a-af09-97634e62d961" (UID: "0885f161-820e-474a-af09-97634e62d961"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:49:09.604303 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.604270    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0885f161-820e-474a-af09-97634e62d961-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:49:09.604303 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.604297    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tq2w\" (UniqueName: \"kubernetes.io/projected/0885f161-820e-474a-af09-97634e62d961-kube-api-access-8tq2w\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:49:09.604303 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.604311    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0885f161-820e-474a-af09-97634e62d961-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:49:09.801259 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.801223    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"]
Sep 15 11:49:09.805441 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.805412    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-baee2b93/e2e-lifecycle-model-baee2b93-kserve-7c57d9db79-gstz6"]
Sep 15 11:49:09.873761 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:09.873729    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0885f161-820e-474a-af09-97634e62d961" path="/var/lib/kubelet/pods/0885f161-820e-474a-af09-97634e62d961/volumes"
Sep 15 11:49:17.649048 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.649010    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"]
Sep 15 11:49:17.649439 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.649396    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffcaffb2-0724-44a7-b998-d1b0b0dba00e" containerName="payload-pre-processing"
Sep 15 11:49:17.649439 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.649409    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffcaffb2-0724-44a7-b998-d1b0b0dba00e" containerName="payload-pre-processing"
Sep 15 11:49:17.649439 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.649420    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0885f161-820e-474a-af09-97634e62d961" containerName="main"
Sep 15 11:49:17.649439 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.649426    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0885f161-820e-474a-af09-97634e62d961" containerName="main"
Sep 15 11:49:17.649579 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.649487    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffcaffb2-0724-44a7-b998-d1b0b0dba00e" containerName="payload-pre-processing"
Sep 15 11:49:17.649579 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.649500    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0885f161-820e-474a-af09-97634e62d961" containerName="main"
Sep 15 11:49:17.653728 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.653712    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:17.655738 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.655712    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b8fbdd\"/\"distinct-2-w4-b8fbdd-kserve-self-signed-certs\""
Sep 15 11:49:17.660923 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.660706    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"]
Sep 15 11:49:17.672937 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.672914    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-dshm\") pod \"distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") " pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:17.673089 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.672947    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zq87\" (UniqueName: \"kubernetes.io/projected/616b0cff-38bc-4a86-ac54-e6b83827bea3-kube-api-access-4zq87\") pod \"distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") " pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:17.673089 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.672973    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/616b0cff-38bc-4a86-ac54-e6b83827bea3-tls-certs\") pod \"distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") " pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:17.673089 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.673040    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-model-cache\") pod \"distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") " pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:17.673284 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.673105    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-home\") pod \"distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") " pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:17.673284 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.673161    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-tmp-dir\") pod \"distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") " pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:17.774257 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.774224    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-dshm\") pod \"distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") " pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:17.774257 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.774260    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zq87\" (UniqueName: \"kubernetes.io/projected/616b0cff-38bc-4a86-ac54-e6b83827bea3-kube-api-access-4zq87\") pod \"distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") " pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:17.774503 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.774285    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/616b0cff-38bc-4a86-ac54-e6b83827bea3-tls-certs\") pod \"distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") " pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:17.774503 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.774307    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-model-cache\") pod \"distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") " pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:17.774503 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.774330    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-home\") pod \"distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") " pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:17.774503 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.774377    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-tmp-dir\") pod \"distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") " pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:17.774754 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.774729    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-home\") pod \"distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") " pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:17.774823 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.774758    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-model-cache\") pod \"distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") " pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:17.774823 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.774809    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-tmp-dir\") pod \"distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") " pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:17.776512 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.776493    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-dshm\") pod \"distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") " pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:17.776690 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.776675    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/616b0cff-38bc-4a86-ac54-e6b83827bea3-tls-certs\") pod \"distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") " pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:17.783433 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.783413    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zq87\" (UniqueName: \"kubernetes.io/projected/616b0cff-38bc-4a86-ac54-e6b83827bea3-kube-api-access-4zq87\") pod \"distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") " pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:17.963888 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:17.963816    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:18.236043 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:18.235869    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"]
Sep 15 11:49:18.238694 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:49:18.238664    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod616b0cff_38bc_4a86_ac54_e6b83827bea3.slice/crio-43b8595efde5bdb25febe040503a36b8c20afd0e2e86962fbc97423faeb78d8b WatchSource:0}: Error finding container 43b8595efde5bdb25febe040503a36b8c20afd0e2e86962fbc97423faeb78d8b: Status 404 returned error can't find the container with id 43b8595efde5bdb25febe040503a36b8c20afd0e2e86962fbc97423faeb78d8b
Sep 15 11:49:18.528667 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:18.528585    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5" event={"ID":"616b0cff-38bc-4a86-ac54-e6b83827bea3","Type":"ContainerStarted","Data":"9232f3f361ce3d723d7dfa2b3cb2ee9a36690427e87add2bc2fc3e74ba82ff95"}
Sep 15 11:49:18.528667 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:18.528632    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5" event={"ID":"616b0cff-38bc-4a86-ac54-e6b83827bea3","Type":"ContainerStarted","Data":"43b8595efde5bdb25febe040503a36b8c20afd0e2e86962fbc97423faeb78d8b"}
Sep 15 11:49:18.545637 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:18.545590    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5" podStartSLOduration=1.5455748969999998 podStartE2EDuration="1.545574897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:49:18.54368593 +0000 UTC m=+1457.206181111" watchObservedRunningTime="2026-09-15 11:49:18.545574897 +0000 UTC m=+1457.208070077"
Sep 15 11:49:27.964973 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:27.964926    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:27.964973 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:27.964980    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:27.977641 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:27.977617    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:28.352836 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:28.352800    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:49:28.586570 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:28.586536    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:49:54.846586 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:49:54.846551    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:50:22.857449 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:22.857410    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:50:46.355423 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:46.355384    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"]
Sep 15 11:50:46.359355 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:46.359325    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"
Sep 15 11:50:46.361854 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:46.361832    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-bcb732-serving-cert\""
Sep 15 11:50:46.375955 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:46.375892    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"]
Sep 15 11:50:46.468423 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:46.468392    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04395e5c-fd35-440b-b3d8-cbcff32eae77-maas-api-tls\") pod \"maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2\" (UID: \"04395e5c-fd35-440b-b3d8-cbcff32eae77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"
Sep 15 11:50:46.468618 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:46.468433    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc2dr\" (UniqueName: \"kubernetes.io/projected/04395e5c-fd35-440b-b3d8-cbcff32eae77-kube-api-access-cc2dr\") pod \"maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2\" (UID: \"04395e5c-fd35-440b-b3d8-cbcff32eae77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"
Sep 15 11:50:46.468618 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:46.468548    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04395e5c-fd35-440b-b3d8-cbcff32eae77-metrics-certs\") pod \"maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2\" (UID: \"04395e5c-fd35-440b-b3d8-cbcff32eae77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"
Sep 15 11:50:46.468618 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:46.468587    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04395e5c-fd35-440b-b3d8-cbcff32eae77-openshift-service-ca\") pod \"maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2\" (UID: \"04395e5c-fd35-440b-b3d8-cbcff32eae77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"
Sep 15 11:50:46.570052 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:46.570017    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04395e5c-fd35-440b-b3d8-cbcff32eae77-openshift-service-ca\") pod \"maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2\" (UID: \"04395e5c-fd35-440b-b3d8-cbcff32eae77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"
Sep 15 11:50:46.570273 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:46.570112    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04395e5c-fd35-440b-b3d8-cbcff32eae77-maas-api-tls\") pod \"maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2\" (UID: \"04395e5c-fd35-440b-b3d8-cbcff32eae77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"
Sep 15 11:50:46.570273 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:46.570137    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc2dr\" (UniqueName: \"kubernetes.io/projected/04395e5c-fd35-440b-b3d8-cbcff32eae77-kube-api-access-cc2dr\") pod \"maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2\" (UID: \"04395e5c-fd35-440b-b3d8-cbcff32eae77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"
Sep 15 11:50:46.570273 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:46.570196    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04395e5c-fd35-440b-b3d8-cbcff32eae77-metrics-certs\") pod \"maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2\" (UID: \"04395e5c-fd35-440b-b3d8-cbcff32eae77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"
Sep 15 11:50:46.570273 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:50:46.570264    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-serving-cert: secret "maas-api-e2e-api-a-bcb732-serving-cert" not found
Sep 15 11:50:46.570490 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:50:46.570348    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04395e5c-fd35-440b-b3d8-cbcff32eae77-maas-api-tls podName:04395e5c-fd35-440b-b3d8-cbcff32eae77 nodeName:}" failed. No retries permitted until 2026-09-15 11:50:47.070328199 +0000 UTC m=+1545.732823358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/04395e5c-fd35-440b-b3d8-cbcff32eae77-maas-api-tls") pod "maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2" (UID: "04395e5c-fd35-440b-b3d8-cbcff32eae77") : secret "maas-api-e2e-api-a-bcb732-serving-cert" not found
Sep 15 11:50:46.570763 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:46.570740    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04395e5c-fd35-440b-b3d8-cbcff32eae77-openshift-service-ca\") pod \"maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2\" (UID: \"04395e5c-fd35-440b-b3d8-cbcff32eae77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"
Sep 15 11:50:46.572612 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:46.572595    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04395e5c-fd35-440b-b3d8-cbcff32eae77-metrics-certs\") pod \"maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2\" (UID: \"04395e5c-fd35-440b-b3d8-cbcff32eae77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"
Sep 15 11:50:46.579296 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:46.579272    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc2dr\" (UniqueName: \"kubernetes.io/projected/04395e5c-fd35-440b-b3d8-cbcff32eae77-kube-api-access-cc2dr\") pod \"maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2\" (UID: \"04395e5c-fd35-440b-b3d8-cbcff32eae77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"
Sep 15 11:50:47.074170 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:47.074133    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04395e5c-fd35-440b-b3d8-cbcff32eae77-maas-api-tls\") pod \"maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2\" (UID: \"04395e5c-fd35-440b-b3d8-cbcff32eae77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"
Sep 15 11:50:47.076485 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:47.076462    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04395e5c-fd35-440b-b3d8-cbcff32eae77-maas-api-tls\") pod \"maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2\" (UID: \"04395e5c-fd35-440b-b3d8-cbcff32eae77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"
Sep 15 11:50:47.271877 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:47.271840    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"
Sep 15 11:50:47.543203 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:47.543176    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"]
Sep 15 11:50:47.544806 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:50:47.544771    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04395e5c_fd35_440b_b3d8_cbcff32eae77.slice/crio-fe8432ba1f3f9f36e6eb2735b3a4a98202b8d9b4d42cf022346fc746f61aa8aa WatchSource:0}: Error finding container fe8432ba1f3f9f36e6eb2735b3a4a98202b8d9b4d42cf022346fc746f61aa8aa: Status 404 returned error can't find the container with id fe8432ba1f3f9f36e6eb2735b3a4a98202b8d9b4d42cf022346fc746f61aa8aa
Sep 15 11:50:47.546196 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:47.546179    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 11:50:47.965216 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:47.965174    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2" event={"ID":"04395e5c-fd35-440b-b3d8-cbcff32eae77","Type":"ContainerStarted","Data":"fe8432ba1f3f9f36e6eb2735b3a4a98202b8d9b4d42cf022346fc746f61aa8aa"}
Sep 15 11:50:48.971598 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:48.971565    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2" event={"ID":"04395e5c-fd35-440b-b3d8-cbcff32eae77","Type":"ContainerStarted","Data":"1784eb0f82de9e913e1c72488cb59483e060c19a31c1c2acfef605881c90ac9d"}
Sep 15 11:50:48.971960 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:48.971744    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"
Sep 15 11:50:48.988291 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:48.988246    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2" podStartSLOduration=2.376786301 podStartE2EDuration="2.98823272s" totalImagesPullingTime="611.446419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:50:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:50:48.987598923 +0000 UTC m=+1547.650094125" watchObservedRunningTime="2026-09-15 11:50:48.98823272 +0000 UTC m=+1547.650727900"
Sep 15 11:50:54.982188 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:50:54.982161    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"
Sep 15 11:51:08.758458 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:08.758418    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:51:17.813696 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:17.813658    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm"]
Sep 15 11:51:17.817138 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:17.817120    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm"
Sep 15 11:51:17.819171 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:17.819146    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2f0e97\""
Sep 15 11:51:17.828788 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:17.828765    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm"]
Sep 15 11:51:17.856904 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:17.856871    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q"]
Sep 15 11:51:17.860205 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:17.860189    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q"
Sep 15 11:51:17.862231 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:17.862207    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2f0e97-dockercfg-5cxjl\""
Sep 15 11:51:17.875687 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:17.875660    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q"]
Sep 15 11:51:17.973969 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:17.973929    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htrdk\" (UniqueName: \"kubernetes.io/projected/24664cd7-3eb1-4993-82d1-721445730d2a-kube-api-access-htrdk\") pod \"payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm\" (UID: \"24664cd7-3eb1-4993-82d1-721445730d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm"
Sep 15 11:51:17.974179 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:17.973982    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pgf8\" (UniqueName: \"kubernetes.io/projected/3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3-kube-api-access-7pgf8\") pod \"payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q\" (UID: \"3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3\") " pod="openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q"
Sep 15 11:51:17.974179 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:17.974025    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q\" (UID: \"3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3\") " pod="openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q"
Sep 15 11:51:17.974179 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:17.974077    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24664cd7-3eb1-4993-82d1-721445730d2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm\" (UID: \"24664cd7-3eb1-4993-82d1-721445730d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm"
Sep 15 11:51:18.075534 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:18.075439    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htrdk\" (UniqueName: \"kubernetes.io/projected/24664cd7-3eb1-4993-82d1-721445730d2a-kube-api-access-htrdk\") pod \"payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm\" (UID: \"24664cd7-3eb1-4993-82d1-721445730d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm"
Sep 15 11:51:18.075534 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:18.075489    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pgf8\" (UniqueName: \"kubernetes.io/projected/3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3-kube-api-access-7pgf8\") pod \"payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q\" (UID: \"3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3\") " pod="openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q"
Sep 15 11:51:18.075534 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:18.075517    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q\" (UID: \"3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3\") " pod="openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q"
Sep 15 11:51:18.075827 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:18.075542    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24664cd7-3eb1-4993-82d1-721445730d2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm\" (UID: \"24664cd7-3eb1-4993-82d1-721445730d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm"
Sep 15 11:51:18.076281 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:18.076264    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24664cd7-3eb1-4993-82d1-721445730d2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm\" (UID: \"24664cd7-3eb1-4993-82d1-721445730d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm"
Sep 15 11:51:18.076328 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:18.076267    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q\" (UID: \"3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3\") " pod="openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q"
Sep 15 11:51:18.083402 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:18.083377    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pgf8\" (UniqueName: \"kubernetes.io/projected/3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3-kube-api-access-7pgf8\") pod \"payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q\" (UID: \"3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3\") " pod="openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q"
Sep 15 11:51:18.083677 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:18.083657    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htrdk\" (UniqueName: \"kubernetes.io/projected/24664cd7-3eb1-4993-82d1-721445730d2a-kube-api-access-htrdk\") pod \"payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm\" (UID: \"24664cd7-3eb1-4993-82d1-721445730d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm"
Sep 15 11:51:18.128510 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:18.128488    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm"
Sep 15 11:51:18.171040 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:18.171000    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q"
Sep 15 11:51:18.421866 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:18.421838    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm"]
Sep 15 11:51:18.424616 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:51:18.424586    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24664cd7_3eb1_4993_82d1_721445730d2a.slice/crio-0415e8ed3c56214f85e8a1a3e8bd73a61b1793e9e8bbb14fa8da6a6dcfbdd108 WatchSource:0}: Error finding container 0415e8ed3c56214f85e8a1a3e8bd73a61b1793e9e8bbb14fa8da6a6dcfbdd108: Status 404 returned error can't find the container with id 0415e8ed3c56214f85e8a1a3e8bd73a61b1793e9e8bbb14fa8da6a6dcfbdd108
Sep 15 11:51:18.475674 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:18.475584    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q"]
Sep 15 11:51:18.478030 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:51:18.478007    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed055e1_7dba_40a8_9e9e_e5d76b55a7b3.slice/crio-a8455cecfa9fdc86734c09a0756742a7e39436db802a9b8f85d96f3af0d514dc WatchSource:0}: Error finding container a8455cecfa9fdc86734c09a0756742a7e39436db802a9b8f85d96f3af0d514dc: Status 404 returned error can't find the container with id a8455cecfa9fdc86734c09a0756742a7e39436db802a9b8f85d96f3af0d514dc
Sep 15 11:51:19.121007 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:19.120967    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm" event={"ID":"24664cd7-3eb1-4993-82d1-721445730d2a","Type":"ContainerStarted","Data":"bdc4f8f8caec475321338aec70f4605045457340e06c2255af8fba9d9eb85155"}
Sep 15 11:51:19.121503 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:19.121031    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm" event={"ID":"24664cd7-3eb1-4993-82d1-721445730d2a","Type":"ContainerStarted","Data":"0415e8ed3c56214f85e8a1a3e8bd73a61b1793e9e8bbb14fa8da6a6dcfbdd108"}
Sep 15 11:51:19.121503 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:19.121161    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm"
Sep 15 11:51:19.122607 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:19.122588    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q" event={"ID":"3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3","Type":"ContainerStarted","Data":"b721a0f7a4b5abea73fe4bd5b0766df7448400e4df76a5f37de19e7a0fead9e3"}
Sep 15 11:51:19.122714 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:19.122620    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q" event={"ID":"3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3","Type":"ContainerStarted","Data":"a8455cecfa9fdc86734c09a0756742a7e39436db802a9b8f85d96f3af0d514dc"}
Sep 15 11:51:19.122781 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:19.122715    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q"
Sep 15 11:51:19.143432 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:19.143385    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm" podStartSLOduration=2.143371591 podStartE2EDuration="2.143371591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:51:19.141998137 +0000 UTC m=+1577.804493341" watchObservedRunningTime="2026-09-15 11:51:19.143371591 +0000 UTC m=+1577.805866771"
Sep 15 11:51:19.163136 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:19.163084    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q" podStartSLOduration=2.163039235 podStartE2EDuration="2.163039235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:51:19.162880446 +0000 UTC m=+1577.825375647" watchObservedRunningTime="2026-09-15 11:51:19.163039235 +0000 UTC m=+1577.825534421"
Sep 15 11:51:22.014089 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:22.014037    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"]
Sep 15 11:51:30.133517 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:30.133480    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q"
Sep 15 11:51:30.133930 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:30.133799    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm"
Sep 15 11:51:33.033498 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.033461    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"]
Sep 15 11:51:33.033961 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.033676    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5" podUID="ad5f1805-33e1-4b10-9b1b-db4099ea5505" containerName="maas-api" containerID="cri-o://ba999c4e25221676245c745eb28c9797aa075c7c368bb3bab989883684d77a79" gracePeriod=30
Sep 15 11:51:33.105262 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.105228    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb"]
Sep 15 11:51:33.105526 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.105497    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb" podUID="e76a47bf-2aea-4ddb-a82a-209bee90a254" containerName="payload-processing" containerID="cri-o://4542f7a562faed3632c83a96fa87496515fbc1ac0b25c2ef8df8065d3d47569c" gracePeriod=30
Sep 15 11:51:33.196524 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.196491    2562 generic.go:356] "Generic (PLEG): container finished" podID="ad5f1805-33e1-4b10-9b1b-db4099ea5505" containerID="ba999c4e25221676245c745eb28c9797aa075c7c368bb3bab989883684d77a79" exitCode=0
Sep 15 11:51:33.196684 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.196565    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5" event={"ID":"ad5f1805-33e1-4b10-9b1b-db4099ea5505","Type":"ContainerDied","Data":"ba999c4e25221676245c745eb28c9797aa075c7c368bb3bab989883684d77a79"}
Sep 15 11:51:33.377988 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.377964    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"
Sep 15 11:51:33.380832 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.380812    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb"
Sep 15 11:51:33.404666 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.404320    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq8tk\" (UniqueName: \"kubernetes.io/projected/ad5f1805-33e1-4b10-9b1b-db4099ea5505-kube-api-access-tq8tk\") pod \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\" (UID: \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\") "
Sep 15 11:51:33.404666 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.404396    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c46nm\" (UniqueName: \"kubernetes.io/projected/e76a47bf-2aea-4ddb-a82a-209bee90a254-kube-api-access-c46nm\") pod \"e76a47bf-2aea-4ddb-a82a-209bee90a254\" (UID: \"e76a47bf-2aea-4ddb-a82a-209bee90a254\") "
Sep 15 11:51:33.404666 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.404476    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76a47bf-2aea-4ddb-a82a-209bee90a254-plugins-config-volume\") pod \"e76a47bf-2aea-4ddb-a82a-209bee90a254\" (UID: \"e76a47bf-2aea-4ddb-a82a-209bee90a254\") "
Sep 15 11:51:33.404666 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.404538    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5f1805-33e1-4b10-9b1b-db4099ea5505-openshift-service-ca\") pod \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\" (UID: \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\") "
Sep 15 11:51:33.404666 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.404570    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad5f1805-33e1-4b10-9b1b-db4099ea5505-metrics-certs\") pod \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\" (UID: \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\") "
Sep 15 11:51:33.404666 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.404620    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad5f1805-33e1-4b10-9b1b-db4099ea5505-maas-api-tls\") pod \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\" (UID: \"ad5f1805-33e1-4b10-9b1b-db4099ea5505\") "
Sep 15 11:51:33.406158 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.404992    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad5f1805-33e1-4b10-9b1b-db4099ea5505-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ad5f1805-33e1-4b10-9b1b-db4099ea5505" (UID: "ad5f1805-33e1-4b10-9b1b-db4099ea5505"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:51:33.406158 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.406119    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76a47bf-2aea-4ddb-a82a-209bee90a254-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e76a47bf-2aea-4ddb-a82a-209bee90a254" (UID: "e76a47bf-2aea-4ddb-a82a-209bee90a254"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:51:33.407265 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.407238    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5f1805-33e1-4b10-9b1b-db4099ea5505-kube-api-access-tq8tk" (OuterVolumeSpecName: "kube-api-access-tq8tk") pod "ad5f1805-33e1-4b10-9b1b-db4099ea5505" (UID: "ad5f1805-33e1-4b10-9b1b-db4099ea5505"). InnerVolumeSpecName "kube-api-access-tq8tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:51:33.407389 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.407270    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76a47bf-2aea-4ddb-a82a-209bee90a254-kube-api-access-c46nm" (OuterVolumeSpecName: "kube-api-access-c46nm") pod "e76a47bf-2aea-4ddb-a82a-209bee90a254" (UID: "e76a47bf-2aea-4ddb-a82a-209bee90a254"). InnerVolumeSpecName "kube-api-access-c46nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:51:33.407389 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.407368    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad5f1805-33e1-4b10-9b1b-db4099ea5505-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ad5f1805-33e1-4b10-9b1b-db4099ea5505" (UID: "ad5f1805-33e1-4b10-9b1b-db4099ea5505"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:51:33.408035 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.408011    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad5f1805-33e1-4b10-9b1b-db4099ea5505-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ad5f1805-33e1-4b10-9b1b-db4099ea5505" (UID: "ad5f1805-33e1-4b10-9b1b-db4099ea5505"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:51:33.506187 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.506150    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5f1805-33e1-4b10-9b1b-db4099ea5505-openshift-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:51:33.506187 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.506181    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad5f1805-33e1-4b10-9b1b-db4099ea5505-metrics-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:51:33.506187 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.506190    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad5f1805-33e1-4b10-9b1b-db4099ea5505-maas-api-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:51:33.506418 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.506199    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq8tk\" (UniqueName: \"kubernetes.io/projected/ad5f1805-33e1-4b10-9b1b-db4099ea5505-kube-api-access-tq8tk\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:51:33.506418 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.506208    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c46nm\" (UniqueName: \"kubernetes.io/projected/e76a47bf-2aea-4ddb-a82a-209bee90a254-kube-api-access-c46nm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:51:33.506418 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:33.506216    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76a47bf-2aea-4ddb-a82a-209bee90a254-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:51:34.203822 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:34.203738    2562 generic.go:356] "Generic (PLEG): container finished" podID="e76a47bf-2aea-4ddb-a82a-209bee90a254" containerID="4542f7a562faed3632c83a96fa87496515fbc1ac0b25c2ef8df8065d3d47569c" exitCode=0
Sep 15 11:51:34.203822 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:34.203788    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb"
Sep 15 11:51:34.204346 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:34.203815    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb" event={"ID":"e76a47bf-2aea-4ddb-a82a-209bee90a254","Type":"ContainerDied","Data":"4542f7a562faed3632c83a96fa87496515fbc1ac0b25c2ef8df8065d3d47569c"}
Sep 15 11:51:34.204346 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:34.203869    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb" event={"ID":"e76a47bf-2aea-4ddb-a82a-209bee90a254","Type":"ContainerDied","Data":"83cd9d904c72d395baa6eb07831c29d278fd5663933caabb8cd89df35ff7637f"}
Sep 15 11:51:34.204346 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:34.203895    2562 scope.go:117] "RemoveContainer" containerID="4542f7a562faed3632c83a96fa87496515fbc1ac0b25c2ef8df8065d3d47569c"
Sep 15 11:51:34.205413 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:34.205395    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"
Sep 15 11:51:34.205499 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:34.205431    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5" event={"ID":"ad5f1805-33e1-4b10-9b1b-db4099ea5505","Type":"ContainerDied","Data":"98072d0f6d8c2d13d7fce51740478ac84c612a001d7bedbfcc02befaf7969df1"}
Sep 15 11:51:34.212784 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:34.212767    2562 scope.go:117] "RemoveContainer" containerID="4542f7a562faed3632c83a96fa87496515fbc1ac0b25c2ef8df8065d3d47569c"
Sep 15 11:51:34.213044 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:51:34.213023    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4542f7a562faed3632c83a96fa87496515fbc1ac0b25c2ef8df8065d3d47569c\": container with ID starting with 4542f7a562faed3632c83a96fa87496515fbc1ac0b25c2ef8df8065d3d47569c not found: ID does not exist" containerID="4542f7a562faed3632c83a96fa87496515fbc1ac0b25c2ef8df8065d3d47569c"
Sep 15 11:51:34.213128 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:34.213069    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4542f7a562faed3632c83a96fa87496515fbc1ac0b25c2ef8df8065d3d47569c"} err="failed to get container status \"4542f7a562faed3632c83a96fa87496515fbc1ac0b25c2ef8df8065d3d47569c\": rpc error: code = NotFound desc = could not find container \"4542f7a562faed3632c83a96fa87496515fbc1ac0b25c2ef8df8065d3d47569c\": container with ID starting with 4542f7a562faed3632c83a96fa87496515fbc1ac0b25c2ef8df8065d3d47569c not found: ID does not exist"
Sep 15 11:51:34.213128 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:34.213093    2562 scope.go:117] "RemoveContainer" containerID="ba999c4e25221676245c745eb28c9797aa075c7c368bb3bab989883684d77a79"
Sep 15 11:51:34.222588 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:34.222570    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"]
Sep 15 11:51:34.227517 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:34.227495    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6e996c-6768d87bbc-248k5"]
Sep 15 11:51:34.242379 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:34.242357    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb"]
Sep 15 11:51:34.246435 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:34.246415    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6e996c-579b598687-nwdqb"]
Sep 15 11:51:35.874483 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:35.874446    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad5f1805-33e1-4b10-9b1b-db4099ea5505" path="/var/lib/kubelet/pods/ad5f1805-33e1-4b10-9b1b-db4099ea5505/volumes"
Sep 15 11:51:35.874874 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:35.874811    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e76a47bf-2aea-4ddb-a82a-209bee90a254" path="/var/lib/kubelet/pods/e76a47bf-2aea-4ddb-a82a-209bee90a254/volumes"
Sep 15 11:51:37.033143 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.033101    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67" podUID="3a04793a-5f2b-4920-8acb-dd0639c1b04f" containerName="main" containerID="cri-o://a83535ed9a0786e8678faa71656b701a11199d7f38c996cc444f0c84c86ff252" gracePeriod=45
Sep 15 11:51:37.234588 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.234548    2562 generic.go:356] "Generic (PLEG): container finished" podID="3a04793a-5f2b-4920-8acb-dd0639c1b04f" containerID="a83535ed9a0786e8678faa71656b701a11199d7f38c996cc444f0c84c86ff252" exitCode=0
Sep 15 11:51:37.234758 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.234633    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67" event={"ID":"3a04793a-5f2b-4920-8acb-dd0639c1b04f","Type":"ContainerDied","Data":"a83535ed9a0786e8678faa71656b701a11199d7f38c996cc444f0c84c86ff252"}
Sep 15 11:51:37.299892 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.299864    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:51:37.338993 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.338965    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpnrr\" (UniqueName: \"kubernetes.io/projected/3a04793a-5f2b-4920-8acb-dd0639c1b04f-kube-api-access-gpnrr\") pod \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") "
Sep 15 11:51:37.339230 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.339033    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-tmp-dir\") pod \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") "
Sep 15 11:51:37.339230 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.339113    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-model-cache\") pod \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") "
Sep 15 11:51:37.339230 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.339143    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-dshm\") pod \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") "
Sep 15 11:51:37.339230 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.339222    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a04793a-5f2b-4920-8acb-dd0639c1b04f-tls-certs\") pod \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") "
Sep 15 11:51:37.339461 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.339253    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-home\") pod \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\" (UID: \"3a04793a-5f2b-4920-8acb-dd0639c1b04f\") "
Sep 15 11:51:37.339461 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.339379    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a04793a-5f2b-4920-8acb-dd0639c1b04f" (UID: "3a04793a-5f2b-4920-8acb-dd0639c1b04f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:51:37.339595 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.339570    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-home" (OuterVolumeSpecName: "home") pod "3a04793a-5f2b-4920-8acb-dd0639c1b04f" (UID: "3a04793a-5f2b-4920-8acb-dd0639c1b04f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:51:37.339727 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.339621    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:51:37.339836 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.339817    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a04793a-5f2b-4920-8acb-dd0639c1b04f" (UID: "3a04793a-5f2b-4920-8acb-dd0639c1b04f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:51:37.342026 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.341986    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a04793a-5f2b-4920-8acb-dd0639c1b04f-kube-api-access-gpnrr" (OuterVolumeSpecName: "kube-api-access-gpnrr") pod "3a04793a-5f2b-4920-8acb-dd0639c1b04f" (UID: "3a04793a-5f2b-4920-8acb-dd0639c1b04f"). InnerVolumeSpecName "kube-api-access-gpnrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:51:37.342182 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.342030    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a04793a-5f2b-4920-8acb-dd0639c1b04f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a04793a-5f2b-4920-8acb-dd0639c1b04f" (UID: "3a04793a-5f2b-4920-8acb-dd0639c1b04f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:51:37.342182 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.342158    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-dshm" (OuterVolumeSpecName: "dshm") pod "3a04793a-5f2b-4920-8acb-dd0639c1b04f" (UID: "3a04793a-5f2b-4920-8acb-dd0639c1b04f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:51:37.441036 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.441002    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:51:37.441036 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.441034    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:51:37.441036 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.441043    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a04793a-5f2b-4920-8acb-dd0639c1b04f-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:51:37.441281 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.441050    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a04793a-5f2b-4920-8acb-dd0639c1b04f-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:51:37.441281 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:37.441088    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpnrr\" (UniqueName: \"kubernetes.io/projected/3a04793a-5f2b-4920-8acb-dd0639c1b04f-kube-api-access-gpnrr\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:51:38.241328 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:38.241247    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67" event={"ID":"3a04793a-5f2b-4920-8acb-dd0639c1b04f","Type":"ContainerDied","Data":"2a8f83a08cb6d0efec34a97aa422275c51a0e1bdb98ed8607a2e6b67e25c67ac"}
Sep 15 11:51:38.241328 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:38.241258    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"
Sep 15 11:51:38.241328 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:38.241299    2562 scope.go:117] "RemoveContainer" containerID="a83535ed9a0786e8678faa71656b701a11199d7f38c996cc444f0c84c86ff252"
Sep 15 11:51:38.256345 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:38.256313    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"]
Sep 15 11:51:38.259771 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:38.259746    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6e996c/simulator-w0-6e996c-kserve-5fdccf9d8f-nrv67"]
Sep 15 11:51:39.874572 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:39.874538    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a04793a-5f2b-4920-8acb-dd0639c1b04f" path="/var/lib/kubelet/pods/3a04793a-5f2b-4920-8acb-dd0639c1b04f/volumes"
Sep 15 11:51:49.086563 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:49.086525    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"]
Sep 15 11:51:49.087207 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:49.086815    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2" podUID="04395e5c-fd35-440b-b3d8-cbcff32eae77" containerName="maas-api" containerID="cri-o://1784eb0f82de9e913e1c72488cb59483e060c19a31c1c2acfef605881c90ac9d" gracePeriod=30
Sep 15 11:51:49.299633 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:49.299604    2562 generic.go:356] "Generic (PLEG): container finished" podID="04395e5c-fd35-440b-b3d8-cbcff32eae77" containerID="1784eb0f82de9e913e1c72488cb59483e060c19a31c1c2acfef605881c90ac9d" exitCode=0
Sep 15 11:51:49.299785 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:49.299680    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2" event={"ID":"04395e5c-fd35-440b-b3d8-cbcff32eae77","Type":"ContainerDied","Data":"1784eb0f82de9e913e1c72488cb59483e060c19a31c1c2acfef605881c90ac9d"}
Sep 15 11:51:49.338208 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:49.338151    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"
Sep 15 11:51:49.448833 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:49.448802    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04395e5c-fd35-440b-b3d8-cbcff32eae77-maas-api-tls\") pod \"04395e5c-fd35-440b-b3d8-cbcff32eae77\" (UID: \"04395e5c-fd35-440b-b3d8-cbcff32eae77\") "
Sep 15 11:51:49.449025 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:49.448911    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cc2dr\" (UniqueName: \"kubernetes.io/projected/04395e5c-fd35-440b-b3d8-cbcff32eae77-kube-api-access-cc2dr\") pod \"04395e5c-fd35-440b-b3d8-cbcff32eae77\" (UID: \"04395e5c-fd35-440b-b3d8-cbcff32eae77\") "
Sep 15 11:51:49.449025 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:49.448939    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04395e5c-fd35-440b-b3d8-cbcff32eae77-openshift-service-ca\") pod \"04395e5c-fd35-440b-b3d8-cbcff32eae77\" (UID: \"04395e5c-fd35-440b-b3d8-cbcff32eae77\") "
Sep 15 11:51:49.449025 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:49.448989    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04395e5c-fd35-440b-b3d8-cbcff32eae77-metrics-certs\") pod \"04395e5c-fd35-440b-b3d8-cbcff32eae77\" (UID: \"04395e5c-fd35-440b-b3d8-cbcff32eae77\") "
Sep 15 11:51:49.449353 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:49.449327    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04395e5c-fd35-440b-b3d8-cbcff32eae77-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "04395e5c-fd35-440b-b3d8-cbcff32eae77" (UID: "04395e5c-fd35-440b-b3d8-cbcff32eae77"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:51:49.451081 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:49.451025    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04395e5c-fd35-440b-b3d8-cbcff32eae77-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "04395e5c-fd35-440b-b3d8-cbcff32eae77" (UID: "04395e5c-fd35-440b-b3d8-cbcff32eae77"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:51:49.451081 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:49.451042    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04395e5c-fd35-440b-b3d8-cbcff32eae77-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "04395e5c-fd35-440b-b3d8-cbcff32eae77" (UID: "04395e5c-fd35-440b-b3d8-cbcff32eae77"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:51:49.451197 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:49.451146    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04395e5c-fd35-440b-b3d8-cbcff32eae77-kube-api-access-cc2dr" (OuterVolumeSpecName: "kube-api-access-cc2dr") pod "04395e5c-fd35-440b-b3d8-cbcff32eae77" (UID: "04395e5c-fd35-440b-b3d8-cbcff32eae77"). InnerVolumeSpecName "kube-api-access-cc2dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:51:49.550246 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:49.550209    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04395e5c-fd35-440b-b3d8-cbcff32eae77-maas-api-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:51:49.550246 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:49.550242    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc2dr\" (UniqueName: \"kubernetes.io/projected/04395e5c-fd35-440b-b3d8-cbcff32eae77-kube-api-access-cc2dr\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:51:49.550433 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:49.550256    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04395e5c-fd35-440b-b3d8-cbcff32eae77-openshift-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:51:49.550433 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:49.550268    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04395e5c-fd35-440b-b3d8-cbcff32eae77-metrics-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:51:50.305808 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:50.305724    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2" event={"ID":"04395e5c-fd35-440b-b3d8-cbcff32eae77","Type":"ContainerDied","Data":"fe8432ba1f3f9f36e6eb2735b3a4a98202b8d9b4d42cf022346fc746f61aa8aa"}
Sep 15 11:51:50.305808 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:50.305747    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"
Sep 15 11:51:50.305808 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:50.305781    2562 scope.go:117] "RemoveContainer" containerID="1784eb0f82de9e913e1c72488cb59483e060c19a31c1c2acfef605881c90ac9d"
Sep 15 11:51:50.327860 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:50.327833    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"]
Sep 15 11:51:50.334845 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:50.334823    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bcb732-857fbc756d-mtwk2"]
Sep 15 11:51:51.873641 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:51.873609    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04395e5c-fd35-440b-b3d8-cbcff32eae77" path="/var/lib/kubelet/pods/04395e5c-fd35-440b-b3d8-cbcff32eae77/volumes"
Sep 15 11:51:56.275592 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.275506    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f0e97-qhfll"]
Sep 15 11:51:56.276102 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.276081    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e76a47bf-2aea-4ddb-a82a-209bee90a254" containerName="payload-processing"
Sep 15 11:51:56.276195 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.276105    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e76a47bf-2aea-4ddb-a82a-209bee90a254" containerName="payload-processing"
Sep 15 11:51:56.276195 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.276135    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04395e5c-fd35-440b-b3d8-cbcff32eae77" containerName="maas-api"
Sep 15 11:51:56.276195 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.276145    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="04395e5c-fd35-440b-b3d8-cbcff32eae77" containerName="maas-api"
Sep 15 11:51:56.276195 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.276164    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad5f1805-33e1-4b10-9b1b-db4099ea5505" containerName="maas-api"
Sep 15 11:51:56.276195 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.276173    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad5f1805-33e1-4b10-9b1b-db4099ea5505" containerName="maas-api"
Sep 15 11:51:56.276195 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.276195    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a04793a-5f2b-4920-8acb-dd0639c1b04f" containerName="main"
Sep 15 11:51:56.276540 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.276204    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a04793a-5f2b-4920-8acb-dd0639c1b04f" containerName="main"
Sep 15 11:51:56.276540 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.276294    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad5f1805-33e1-4b10-9b1b-db4099ea5505" containerName="maas-api"
Sep 15 11:51:56.276540 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.276312    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="04395e5c-fd35-440b-b3d8-cbcff32eae77" containerName="maas-api"
Sep 15 11:51:56.276540 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.276323    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e76a47bf-2aea-4ddb-a82a-209bee90a254" containerName="payload-processing"
Sep 15 11:51:56.276540 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.276334    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a04793a-5f2b-4920-8acb-dd0639c1b04f" containerName="main"
Sep 15 11:51:56.280803 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.280781    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f0e97-qhfll"
Sep 15 11:51:56.282849 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.282829    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rbxjk\""
Sep 15 11:51:56.286066 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.286024    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f0e97-qhfll"]
Sep 15 11:51:56.407502 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.407466    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0306670-cf0d-4a62-aa38-230c11d9ed36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2f0e97-qhfll\" (UID: \"a0306670-cf0d-4a62-aa38-230c11d9ed36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f0e97-qhfll"
Sep 15 11:51:56.508505 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.508473    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0306670-cf0d-4a62-aa38-230c11d9ed36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2f0e97-qhfll\" (UID: \"a0306670-cf0d-4a62-aa38-230c11d9ed36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f0e97-qhfll"
Sep 15 11:51:56.509033 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.509014    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0306670-cf0d-4a62-aa38-230c11d9ed36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2f0e97-qhfll\" (UID: \"a0306670-cf0d-4a62-aa38-230c11d9ed36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f0e97-qhfll"
Sep 15 11:51:56.592230 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.592210    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f0e97-qhfll"
Sep 15 11:51:56.862923 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:56.862855    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f0e97-qhfll"]
Sep 15 11:51:56.865680 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:51:56.865645    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0306670_cf0d_4a62_aa38_230c11d9ed36.slice/crio-a0fbb4b3451bc2501064af34081011f9795c159eb3082b3f7c65264739c00fdc WatchSource:0}: Error finding container a0fbb4b3451bc2501064af34081011f9795c159eb3082b3f7c65264739c00fdc: Status 404 returned error can't find the container with id a0fbb4b3451bc2501064af34081011f9795c159eb3082b3f7c65264739c00fdc
Sep 15 11:51:57.344849 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:57.344810    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f0e97-qhfll" event={"ID":"a0306670-cf0d-4a62-aa38-230c11d9ed36","Type":"ContainerStarted","Data":"073cd3896b5882a5f84b16bf1bcdd3db2050524687f58de9776608cb1bf5c4f3"}
Sep 15 11:51:57.345254 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:57.344876    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f0e97-qhfll" event={"ID":"a0306670-cf0d-4a62-aa38-230c11d9ed36","Type":"ContainerStarted","Data":"a0fbb4b3451bc2501064af34081011f9795c159eb3082b3f7c65264739c00fdc"}
Sep 15 11:51:57.371556 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:57.371507    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f0e97-qhfll" podStartSLOduration=1.371491818 podStartE2EDuration="1.371491818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:51:57.370295997 +0000 UTC m=+1616.032791198" watchObservedRunningTime="2026-09-15 11:51:57.371491818 +0000 UTC m=+1616.033986998"
Sep 15 11:51:58.350631 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:58.350597    2562 generic.go:356] "Generic (PLEG): container finished" podID="a0306670-cf0d-4a62-aa38-230c11d9ed36" containerID="073cd3896b5882a5f84b16bf1bcdd3db2050524687f58de9776608cb1bf5c4f3" exitCode=0
Sep 15 11:51:58.351024 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:58.350662    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f0e97-qhfll" event={"ID":"a0306670-cf0d-4a62-aa38-230c11d9ed36","Type":"ContainerDied","Data":"073cd3896b5882a5f84b16bf1bcdd3db2050524687f58de9776608cb1bf5c4f3"}
Sep 15 11:51:59.496956 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:59.496929    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f0e97-qhfll"
Sep 15 11:51:59.571434 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:59.571396    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:51:59.636367 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:59.636298    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0306670-cf0d-4a62-aa38-230c11d9ed36-maas-api-service-ca\") pod \"a0306670-cf0d-4a62-aa38-230c11d9ed36\" (UID: \"a0306670-cf0d-4a62-aa38-230c11d9ed36\") "
Sep 15 11:51:59.636681 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:59.636652    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0306670-cf0d-4a62-aa38-230c11d9ed36-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0306670-cf0d-4a62-aa38-230c11d9ed36" (UID: "a0306670-cf0d-4a62-aa38-230c11d9ed36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:51:59.737682 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:51:59.737651    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0306670-cf0d-4a62-aa38-230c11d9ed36-maas-api-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:00.361345 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:00.361306    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f0e97-qhfll" event={"ID":"a0306670-cf0d-4a62-aa38-230c11d9ed36","Type":"ContainerDied","Data":"a0fbb4b3451bc2501064af34081011f9795c159eb3082b3f7c65264739c00fdc"}
Sep 15 11:52:00.361539 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:00.361368    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0fbb4b3451bc2501064af34081011f9795c159eb3082b3f7c65264739c00fdc"
Sep 15 11:52:00.361539 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:00.361383    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f0e97-qhfll"
Sep 15 11:52:02.024590 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:02.024555    2562 scope.go:117] "RemoveContainer" containerID="6dd8bfb6c5937648468ad56ce036107147e456859aaef8388338c0bbc986191a"
Sep 15 11:52:11.953878 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:11.953844    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"]
Sep 15 11:52:11.970032 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:11.969998    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"]
Sep 15 11:52:13.306214 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.306179    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q"]
Sep 15 11:52:13.306694 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.306428    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q" podUID="3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3" containerName="payload-processing" containerID="cri-o://b721a0f7a4b5abea73fe4bd5b0766df7448400e4df76a5f37de19e7a0fead9e3" gracePeriod=30
Sep 15 11:52:13.311742 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.311709    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm"]
Sep 15 11:52:13.311950 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.311925    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm" podUID="24664cd7-3eb1-4993-82d1-721445730d2a" containerName="payload-pre-processing" containerID="cri-o://bdc4f8f8caec475321338aec70f4605045457340e06c2255af8fba9d9eb85155" gracePeriod=30
Sep 15 11:52:13.426416 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.426383    2562 generic.go:356] "Generic (PLEG): container finished" podID="3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3" containerID="b721a0f7a4b5abea73fe4bd5b0766df7448400e4df76a5f37de19e7a0fead9e3" exitCode=0
Sep 15 11:52:13.426533 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.426452    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q" event={"ID":"3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3","Type":"ContainerDied","Data":"b721a0f7a4b5abea73fe4bd5b0766df7448400e4df76a5f37de19e7a0fead9e3"}
Sep 15 11:52:13.492793 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.492760    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f0e97-qhfll"]
Sep 15 11:52:13.498535 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.498509    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f0e97-qhfll"]
Sep 15 11:52:13.589232 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.589201    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q"
Sep 15 11:52:13.591936 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.591915    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm"
Sep 15 11:52:13.762584 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.762549    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24664cd7-3eb1-4993-82d1-721445730d2a-plugins-config-volume\") pod \"24664cd7-3eb1-4993-82d1-721445730d2a\" (UID: \"24664cd7-3eb1-4993-82d1-721445730d2a\") "
Sep 15 11:52:13.762784 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.762596    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htrdk\" (UniqueName: \"kubernetes.io/projected/24664cd7-3eb1-4993-82d1-721445730d2a-kube-api-access-htrdk\") pod \"24664cd7-3eb1-4993-82d1-721445730d2a\" (UID: \"24664cd7-3eb1-4993-82d1-721445730d2a\") "
Sep 15 11:52:13.762784 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.762620    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3-plugins-config-volume\") pod \"3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3\" (UID: \"3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3\") "
Sep 15 11:52:13.762784 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.762682    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pgf8\" (UniqueName: \"kubernetes.io/projected/3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3-kube-api-access-7pgf8\") pod \"3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3\" (UID: \"3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3\") "
Sep 15 11:52:13.763032 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.763007    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24664cd7-3eb1-4993-82d1-721445730d2a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24664cd7-3eb1-4993-82d1-721445730d2a" (UID: "24664cd7-3eb1-4993-82d1-721445730d2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:52:13.763134 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.763108    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3" (UID: "3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:52:13.764853 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.764831    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3-kube-api-access-7pgf8" (OuterVolumeSpecName: "kube-api-access-7pgf8") pod "3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3" (UID: "3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3"). InnerVolumeSpecName "kube-api-access-7pgf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:52:13.764853 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.764844    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24664cd7-3eb1-4993-82d1-721445730d2a-kube-api-access-htrdk" (OuterVolumeSpecName: "kube-api-access-htrdk") pod "24664cd7-3eb1-4993-82d1-721445730d2a" (UID: "24664cd7-3eb1-4993-82d1-721445730d2a"). InnerVolumeSpecName "kube-api-access-htrdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:52:13.864247 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.864219    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:13.864247 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.864244    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pgf8\" (UniqueName: \"kubernetes.io/projected/3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3-kube-api-access-7pgf8\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:13.864408 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.864254    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24664cd7-3eb1-4993-82d1-721445730d2a-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:13.864408 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.864263    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htrdk\" (UniqueName: \"kubernetes.io/projected/24664cd7-3eb1-4993-82d1-721445730d2a-kube-api-access-htrdk\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:13.873949 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:13.873920    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0306670-cf0d-4a62-aa38-230c11d9ed36" path="/var/lib/kubelet/pods/a0306670-cf0d-4a62-aa38-230c11d9ed36/volumes"
Sep 15 11:52:14.432961 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:14.432926    2562 generic.go:356] "Generic (PLEG): container finished" podID="24664cd7-3eb1-4993-82d1-721445730d2a" containerID="bdc4f8f8caec475321338aec70f4605045457340e06c2255af8fba9d9eb85155" exitCode=0
Sep 15 11:52:14.433425 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:14.432981    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm"
Sep 15 11:52:14.433425 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:14.433014    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm" event={"ID":"24664cd7-3eb1-4993-82d1-721445730d2a","Type":"ContainerDied","Data":"bdc4f8f8caec475321338aec70f4605045457340e06c2255af8fba9d9eb85155"}
Sep 15 11:52:14.433425 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:14.433101    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm" event={"ID":"24664cd7-3eb1-4993-82d1-721445730d2a","Type":"ContainerDied","Data":"0415e8ed3c56214f85e8a1a3e8bd73a61b1793e9e8bbb14fa8da6a6dcfbdd108"}
Sep 15 11:52:14.433425 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:14.433141    2562 scope.go:117] "RemoveContainer" containerID="bdc4f8f8caec475321338aec70f4605045457340e06c2255af8fba9d9eb85155"
Sep 15 11:52:14.434588 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:14.434426    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q" event={"ID":"3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3","Type":"ContainerDied","Data":"a8455cecfa9fdc86734c09a0756742a7e39436db802a9b8f85d96f3af0d514dc"}
Sep 15 11:52:14.434588 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:14.434467    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q"
Sep 15 11:52:14.441988 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:14.441966    2562 scope.go:117] "RemoveContainer" containerID="bdc4f8f8caec475321338aec70f4605045457340e06c2255af8fba9d9eb85155"
Sep 15 11:52:14.442362 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:52:14.442339    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc4f8f8caec475321338aec70f4605045457340e06c2255af8fba9d9eb85155\": container with ID starting with bdc4f8f8caec475321338aec70f4605045457340e06c2255af8fba9d9eb85155 not found: ID does not exist" containerID="bdc4f8f8caec475321338aec70f4605045457340e06c2255af8fba9d9eb85155"
Sep 15 11:52:14.442461 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:14.442372    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc4f8f8caec475321338aec70f4605045457340e06c2255af8fba9d9eb85155"} err="failed to get container status \"bdc4f8f8caec475321338aec70f4605045457340e06c2255af8fba9d9eb85155\": rpc error: code = NotFound desc = could not find container \"bdc4f8f8caec475321338aec70f4605045457340e06c2255af8fba9d9eb85155\": container with ID starting with bdc4f8f8caec475321338aec70f4605045457340e06c2255af8fba9d9eb85155 not found: ID does not exist"
Sep 15 11:52:14.442461 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:14.442394    2562 scope.go:117] "RemoveContainer" containerID="b721a0f7a4b5abea73fe4bd5b0766df7448400e4df76a5f37de19e7a0fead9e3"
Sep 15 11:52:14.479850 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:14.479824    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q"]
Sep 15 11:52:14.482725 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:14.482701    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2f0e97-888ddd8b7-4c78q"]
Sep 15 11:52:14.509610 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:14.509588    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm"]
Sep 15 11:52:14.515515 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:14.515495    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2f0e97-686dcbdfc4-mjfbm"]
Sep 15 11:52:15.873646 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:15.873615    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24664cd7-3eb1-4993-82d1-721445730d2a" path="/var/lib/kubelet/pods/24664cd7-3eb1-4993-82d1-721445730d2a/volumes"
Sep 15 11:52:15.874031 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:15.873950    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3" path="/var/lib/kubelet/pods/3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3/volumes"
Sep 15 11:52:20.741660 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:20.741618    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5a8d6f-djscl"]
Sep 15 11:52:20.742183 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:20.741981    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3" containerName="payload-processing"
Sep 15 11:52:20.742183 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:20.741992    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3" containerName="payload-processing"
Sep 15 11:52:20.742183 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:20.742003    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0306670-cf0d-4a62-aa38-230c11d9ed36" containerName="revoke-keys"
Sep 15 11:52:20.742183 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:20.742008    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0306670-cf0d-4a62-aa38-230c11d9ed36" containerName="revoke-keys"
Sep 15 11:52:20.742183 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:20.742018    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24664cd7-3eb1-4993-82d1-721445730d2a" containerName="payload-pre-processing"
Sep 15 11:52:20.742183 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:20.742023    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="24664cd7-3eb1-4993-82d1-721445730d2a" containerName="payload-pre-processing"
Sep 15 11:52:20.742183 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:20.742102    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0306670-cf0d-4a62-aa38-230c11d9ed36" containerName="revoke-keys"
Sep 15 11:52:20.742183 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:20.742112    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ed055e1-7dba-40a8-9e9e-e5d76b55a7b3" containerName="payload-processing"
Sep 15 11:52:20.742183 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:20.742120    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="24664cd7-3eb1-4993-82d1-721445730d2a" containerName="payload-pre-processing"
Sep 15 11:52:20.746140 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:20.746123    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5a8d6f-djscl"
Sep 15 11:52:20.749937 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:20.749918    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rbxjk\""
Sep 15 11:52:20.754771 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:20.754743    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5a8d6f-djscl"]
Sep 15 11:52:20.824356 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:20.824328    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0acd70eb-9b86-48d5-88db-479b78bf566e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5a8d6f-djscl\" (UID: \"0acd70eb-9b86-48d5-88db-479b78bf566e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5a8d6f-djscl"
Sep 15 11:52:20.925444 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:20.925407    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0acd70eb-9b86-48d5-88db-479b78bf566e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5a8d6f-djscl\" (UID: \"0acd70eb-9b86-48d5-88db-479b78bf566e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5a8d6f-djscl"
Sep 15 11:52:20.925990 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:20.925966    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0acd70eb-9b86-48d5-88db-479b78bf566e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5a8d6f-djscl\" (UID: \"0acd70eb-9b86-48d5-88db-479b78bf566e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5a8d6f-djscl"
Sep 15 11:52:21.058035 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:21.057959    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5a8d6f-djscl"
Sep 15 11:52:21.058846 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:21.058827    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:52:21.333889 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:21.333865    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5a8d6f-djscl"]
Sep 15 11:52:21.336372 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:52:21.336337    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0acd70eb_9b86_48d5_88db_479b78bf566e.slice/crio-81737e659141f7fe91c66936e76c7aa5d6087cf4289af6ddd646bca7a707664a WatchSource:0}: Error finding container 81737e659141f7fe91c66936e76c7aa5d6087cf4289af6ddd646bca7a707664a: Status 404 returned error can't find the container with id 81737e659141f7fe91c66936e76c7aa5d6087cf4289af6ddd646bca7a707664a
Sep 15 11:52:21.469111 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:21.469081    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5a8d6f-djscl" event={"ID":"0acd70eb-9b86-48d5-88db-479b78bf566e","Type":"ContainerStarted","Data":"81737e659141f7fe91c66936e76c7aa5d6087cf4289af6ddd646bca7a707664a"}
Sep 15 11:52:22.475280 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:22.475249    2562 generic.go:356] "Generic (PLEG): container finished" podID="0acd70eb-9b86-48d5-88db-479b78bf566e" containerID="7f217ba0b35329e39dbfc22e31578d6df17adf1f11e2653e516ff54f590f8d41" exitCode=0
Sep 15 11:52:22.475664 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:22.475291    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5a8d6f-djscl" event={"ID":"0acd70eb-9b86-48d5-88db-479b78bf566e","Type":"ContainerDied","Data":"7f217ba0b35329e39dbfc22e31578d6df17adf1f11e2653e516ff54f590f8d41"}
Sep 15 11:52:23.344501 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:23.344458    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6"]
Sep 15 11:52:23.344725 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:23.344697    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6" podUID="0e09cdb0-f82f-4a5c-a90a-710de63fbacb" containerName="payload-pre-processing" containerID="cri-o://7ef16dbbc2063ca09974a0b502d50a170acabaf34ed9ccb7f1f4b812c55b8b35" gracePeriod=30
Sep 15 11:52:23.482112 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:23.482073    2562 generic.go:356] "Generic (PLEG): container finished" podID="0e09cdb0-f82f-4a5c-a90a-710de63fbacb" containerID="7ef16dbbc2063ca09974a0b502d50a170acabaf34ed9ccb7f1f4b812c55b8b35" exitCode=0
Sep 15 11:52:23.482542 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:23.482142    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6" event={"ID":"0e09cdb0-f82f-4a5c-a90a-710de63fbacb","Type":"ContainerDied","Data":"7ef16dbbc2063ca09974a0b502d50a170acabaf34ed9ccb7f1f4b812c55b8b35"}
Sep 15 11:52:23.650543 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:23.650515    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5a8d6f-djscl"
Sep 15 11:52:23.654102 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:23.654076    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6"
Sep 15 11:52:23.746425 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:23.746386    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mkpw\" (UniqueName: \"kubernetes.io/projected/0e09cdb0-f82f-4a5c-a90a-710de63fbacb-kube-api-access-8mkpw\") pod \"0e09cdb0-f82f-4a5c-a90a-710de63fbacb\" (UID: \"0e09cdb0-f82f-4a5c-a90a-710de63fbacb\") "
Sep 15 11:52:23.746624 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:23.746435    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e09cdb0-f82f-4a5c-a90a-710de63fbacb-plugins-config-volume\") pod \"0e09cdb0-f82f-4a5c-a90a-710de63fbacb\" (UID: \"0e09cdb0-f82f-4a5c-a90a-710de63fbacb\") "
Sep 15 11:52:23.746624 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:23.746473    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0acd70eb-9b86-48d5-88db-479b78bf566e-maas-api-service-ca\") pod \"0acd70eb-9b86-48d5-88db-479b78bf566e\" (UID: \"0acd70eb-9b86-48d5-88db-479b78bf566e\") "
Sep 15 11:52:23.746839 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:23.746815    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0acd70eb-9b86-48d5-88db-479b78bf566e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0acd70eb-9b86-48d5-88db-479b78bf566e" (UID: "0acd70eb-9b86-48d5-88db-479b78bf566e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:52:23.746906 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:23.746856    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e09cdb0-f82f-4a5c-a90a-710de63fbacb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0e09cdb0-f82f-4a5c-a90a-710de63fbacb" (UID: "0e09cdb0-f82f-4a5c-a90a-710de63fbacb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:52:23.748474 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:23.748451    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e09cdb0-f82f-4a5c-a90a-710de63fbacb-kube-api-access-8mkpw" (OuterVolumeSpecName: "kube-api-access-8mkpw") pod "0e09cdb0-f82f-4a5c-a90a-710de63fbacb" (UID: "0e09cdb0-f82f-4a5c-a90a-710de63fbacb"). InnerVolumeSpecName "kube-api-access-8mkpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:52:23.847853 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:23.847820    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mkpw\" (UniqueName: \"kubernetes.io/projected/0e09cdb0-f82f-4a5c-a90a-710de63fbacb-kube-api-access-8mkpw\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:23.847853 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:23.847847    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e09cdb0-f82f-4a5c-a90a-710de63fbacb-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:23.847853 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:23.847857    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0acd70eb-9b86-48d5-88db-479b78bf566e-maas-api-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:24.489465 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:24.489430    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6"
Sep 15 11:52:24.489933 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:24.489460    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6" event={"ID":"0e09cdb0-f82f-4a5c-a90a-710de63fbacb","Type":"ContainerDied","Data":"a60d669a7c39e595576396a998ff2b4860981020f93b7e91c0164973380199a9"}
Sep 15 11:52:24.489933 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:24.489546    2562 scope.go:117] "RemoveContainer" containerID="7ef16dbbc2063ca09974a0b502d50a170acabaf34ed9ccb7f1f4b812c55b8b35"
Sep 15 11:52:24.490898 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:24.490880    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5a8d6f-djscl" event={"ID":"0acd70eb-9b86-48d5-88db-479b78bf566e","Type":"ContainerDied","Data":"81737e659141f7fe91c66936e76c7aa5d6087cf4289af6ddd646bca7a707664a"}
Sep 15 11:52:24.490954 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:24.490908    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5a8d6f-djscl"
Sep 15 11:52:24.490993 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:24.490911    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81737e659141f7fe91c66936e76c7aa5d6087cf4289af6ddd646bca7a707664a"
Sep 15 11:52:24.508712 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:24.508678    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6"]
Sep 15 11:52:24.512923 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:24.512901    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b8fbdd-57dd7dfb7d-fd7v6"]
Sep 15 11:52:25.875640 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:25.875607    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e09cdb0-f82f-4a5c-a90a-710de63fbacb" path="/var/lib/kubelet/pods/0e09cdb0-f82f-4a5c-a90a-710de63fbacb/volumes"
Sep 15 11:52:26.973529 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:26.973492    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5" podUID="616b0cff-38bc-4a86-ac54-e6b83827bea3" containerName="main" containerID="cri-o://9232f3f361ce3d723d7dfa2b3cb2ee9a36690427e87add2bc2fc3e74ba82ff95" gracePeriod=45
Sep 15 11:52:26.988637 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:26.988600    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r" podUID="cb2913b3-759c-4dcc-bcc4-fdb3311d1356" containerName="main" containerID="cri-o://d4f8ab42df5a97ee6785f886c9f1a49adf3519718a148dd18e2a84bc0cde6bc6" gracePeriod=45
Sep 15 11:52:27.258428 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.258397    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:52:27.261244 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.261225    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:52:27.275659 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.275635    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-tmp-dir\") pod \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") "
Sep 15 11:52:27.275781 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.275673    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-dshm\") pod \"616b0cff-38bc-4a86-ac54-e6b83827bea3\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") "
Sep 15 11:52:27.275781 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.275703    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqp7q\" (UniqueName: \"kubernetes.io/projected/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-kube-api-access-xqp7q\") pod \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") "
Sep 15 11:52:27.275781 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.275728    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-tls-certs\") pod \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") "
Sep 15 11:52:27.275781 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.275751    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zq87\" (UniqueName: \"kubernetes.io/projected/616b0cff-38bc-4a86-ac54-e6b83827bea3-kube-api-access-4zq87\") pod \"616b0cff-38bc-4a86-ac54-e6b83827bea3\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") "
Sep 15 11:52:27.275781 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.275777    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-dshm\") pod \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") "
Sep 15 11:52:27.276504 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.275800    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-model-cache\") pod \"616b0cff-38bc-4a86-ac54-e6b83827bea3\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") "
Sep 15 11:52:27.276504 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.275888    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-home\") pod \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") "
Sep 15 11:52:27.276504 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.275912    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/616b0cff-38bc-4a86-ac54-e6b83827bea3-tls-certs\") pod \"616b0cff-38bc-4a86-ac54-e6b83827bea3\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") "
Sep 15 11:52:27.276504 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.275911    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cb2913b3-759c-4dcc-bcc4-fdb3311d1356" (UID: "cb2913b3-759c-4dcc-bcc4-fdb3311d1356"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:52:27.276504 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.275970    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-tmp-dir\") pod \"616b0cff-38bc-4a86-ac54-e6b83827bea3\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") "
Sep 15 11:52:27.276504 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.276005    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-home\") pod \"616b0cff-38bc-4a86-ac54-e6b83827bea3\" (UID: \"616b0cff-38bc-4a86-ac54-e6b83827bea3\") "
Sep 15 11:52:27.276504 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.276041    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-model-cache\") pod \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\" (UID: \"cb2913b3-759c-4dcc-bcc4-fdb3311d1356\") "
Sep 15 11:52:27.276504 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.276377    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:27.276986 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.276586    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-model-cache" (OuterVolumeSpecName: "model-cache") pod "cb2913b3-759c-4dcc-bcc4-fdb3311d1356" (UID: "cb2913b3-759c-4dcc-bcc4-fdb3311d1356"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:52:27.276986 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.276756    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "616b0cff-38bc-4a86-ac54-e6b83827bea3" (UID: "616b0cff-38bc-4a86-ac54-e6b83827bea3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:52:27.276986 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.276773    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-home" (OuterVolumeSpecName: "home") pod "616b0cff-38bc-4a86-ac54-e6b83827bea3" (UID: "616b0cff-38bc-4a86-ac54-e6b83827bea3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:52:27.276986 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.276952    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-model-cache" (OuterVolumeSpecName: "model-cache") pod "616b0cff-38bc-4a86-ac54-e6b83827bea3" (UID: "616b0cff-38bc-4a86-ac54-e6b83827bea3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:52:27.277238 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.277073    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-home" (OuterVolumeSpecName: "home") pod "cb2913b3-759c-4dcc-bcc4-fdb3311d1356" (UID: "cb2913b3-759c-4dcc-bcc4-fdb3311d1356"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:52:27.279497 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.279457    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/616b0cff-38bc-4a86-ac54-e6b83827bea3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "616b0cff-38bc-4a86-ac54-e6b83827bea3" (UID: "616b0cff-38bc-4a86-ac54-e6b83827bea3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:52:27.281844 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.281717    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb2913b3-759c-4dcc-bcc4-fdb3311d1356" (UID: "cb2913b3-759c-4dcc-bcc4-fdb3311d1356"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:52:27.281844 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.281766    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-kube-api-access-xqp7q" (OuterVolumeSpecName: "kube-api-access-xqp7q") pod "cb2913b3-759c-4dcc-bcc4-fdb3311d1356" (UID: "cb2913b3-759c-4dcc-bcc4-fdb3311d1356"). InnerVolumeSpecName "kube-api-access-xqp7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:52:27.281844 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.281785    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-dshm" (OuterVolumeSpecName: "dshm") pod "616b0cff-38bc-4a86-ac54-e6b83827bea3" (UID: "616b0cff-38bc-4a86-ac54-e6b83827bea3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:52:27.281844 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.281801    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616b0cff-38bc-4a86-ac54-e6b83827bea3-kube-api-access-4zq87" (OuterVolumeSpecName: "kube-api-access-4zq87") pod "616b0cff-38bc-4a86-ac54-e6b83827bea3" (UID: "616b0cff-38bc-4a86-ac54-e6b83827bea3"). InnerVolumeSpecName "kube-api-access-4zq87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:52:27.281844 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.281844    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-dshm" (OuterVolumeSpecName: "dshm") pod "cb2913b3-759c-4dcc-bcc4-fdb3311d1356" (UID: "cb2913b3-759c-4dcc-bcc4-fdb3311d1356"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:52:27.376777 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.376742    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:27.376777 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.376772    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqp7q\" (UniqueName: \"kubernetes.io/projected/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-kube-api-access-xqp7q\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:27.376777 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.376783    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:27.377090 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.376793    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zq87\" (UniqueName: \"kubernetes.io/projected/616b0cff-38bc-4a86-ac54-e6b83827bea3-kube-api-access-4zq87\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:27.377090 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.376802    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:27.377090 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.376810    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:27.377090 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.376817    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:27.377090 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.376824    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/616b0cff-38bc-4a86-ac54-e6b83827bea3-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:27.377090 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.376831    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:27.377090 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.376838    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/616b0cff-38bc-4a86-ac54-e6b83827bea3-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:27.377090 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.376844    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb2913b3-759c-4dcc-bcc4-fdb3311d1356-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:27.508648 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.508566    2562 generic.go:356] "Generic (PLEG): container finished" podID="cb2913b3-759c-4dcc-bcc4-fdb3311d1356" containerID="d4f8ab42df5a97ee6785f886c9f1a49adf3519718a148dd18e2a84bc0cde6bc6" exitCode=0
Sep 15 11:52:27.508648 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.508624    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"
Sep 15 11:52:27.508861 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.508662    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r" event={"ID":"cb2913b3-759c-4dcc-bcc4-fdb3311d1356","Type":"ContainerDied","Data":"d4f8ab42df5a97ee6785f886c9f1a49adf3519718a148dd18e2a84bc0cde6bc6"}
Sep 15 11:52:27.508861 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.508711    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r" event={"ID":"cb2913b3-759c-4dcc-bcc4-fdb3311d1356","Type":"ContainerDied","Data":"ba5e800ffcb3e541bffd5853371010115eebec8ffdad92885f28290210d2b5e7"}
Sep 15 11:52:27.508861 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.508736    2562 scope.go:117] "RemoveContainer" containerID="d4f8ab42df5a97ee6785f886c9f1a49adf3519718a148dd18e2a84bc0cde6bc6"
Sep 15 11:52:27.510101 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.510050    2562 generic.go:356] "Generic (PLEG): container finished" podID="616b0cff-38bc-4a86-ac54-e6b83827bea3" containerID="9232f3f361ce3d723d7dfa2b3cb2ee9a36690427e87add2bc2fc3e74ba82ff95" exitCode=0
Sep 15 11:52:27.510212 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.510101    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"
Sep 15 11:52:27.510212 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.510116    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5" event={"ID":"616b0cff-38bc-4a86-ac54-e6b83827bea3","Type":"ContainerDied","Data":"9232f3f361ce3d723d7dfa2b3cb2ee9a36690427e87add2bc2fc3e74ba82ff95"}
Sep 15 11:52:27.510212 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.510148    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5" event={"ID":"616b0cff-38bc-4a86-ac54-e6b83827bea3","Type":"ContainerDied","Data":"43b8595efde5bdb25febe040503a36b8c20afd0e2e86962fbc97423faeb78d8b"}
Sep 15 11:52:27.519210 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.519188    2562 scope.go:117] "RemoveContainer" containerID="d4f8ab42df5a97ee6785f886c9f1a49adf3519718a148dd18e2a84bc0cde6bc6"
Sep 15 11:52:27.519496 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:52:27.519473    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4f8ab42df5a97ee6785f886c9f1a49adf3519718a148dd18e2a84bc0cde6bc6\": container with ID starting with d4f8ab42df5a97ee6785f886c9f1a49adf3519718a148dd18e2a84bc0cde6bc6 not found: ID does not exist" containerID="d4f8ab42df5a97ee6785f886c9f1a49adf3519718a148dd18e2a84bc0cde6bc6"
Sep 15 11:52:27.519574 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.519506    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f8ab42df5a97ee6785f886c9f1a49adf3519718a148dd18e2a84bc0cde6bc6"} err="failed to get container status \"d4f8ab42df5a97ee6785f886c9f1a49adf3519718a148dd18e2a84bc0cde6bc6\": rpc error: code = NotFound desc = could not find container \"d4f8ab42df5a97ee6785f886c9f1a49adf3519718a148dd18e2a84bc0cde6bc6\": container with ID starting with d4f8ab42df5a97ee6785f886c9f1a49adf3519718a148dd18e2a84bc0cde6bc6 not found: ID does not exist"
Sep 15 11:52:27.519574 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.519532    2562 scope.go:117] "RemoveContainer" containerID="9232f3f361ce3d723d7dfa2b3cb2ee9a36690427e87add2bc2fc3e74ba82ff95"
Sep 15 11:52:27.527105 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.527089    2562 scope.go:117] "RemoveContainer" containerID="9232f3f361ce3d723d7dfa2b3cb2ee9a36690427e87add2bc2fc3e74ba82ff95"
Sep 15 11:52:27.527353 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:52:27.527328    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9232f3f361ce3d723d7dfa2b3cb2ee9a36690427e87add2bc2fc3e74ba82ff95\": container with ID starting with 9232f3f361ce3d723d7dfa2b3cb2ee9a36690427e87add2bc2fc3e74ba82ff95 not found: ID does not exist" containerID="9232f3f361ce3d723d7dfa2b3cb2ee9a36690427e87add2bc2fc3e74ba82ff95"
Sep 15 11:52:27.527421 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.527357    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9232f3f361ce3d723d7dfa2b3cb2ee9a36690427e87add2bc2fc3e74ba82ff95"} err="failed to get container status \"9232f3f361ce3d723d7dfa2b3cb2ee9a36690427e87add2bc2fc3e74ba82ff95\": rpc error: code = NotFound desc = could not find container \"9232f3f361ce3d723d7dfa2b3cb2ee9a36690427e87add2bc2fc3e74ba82ff95\": container with ID starting with 9232f3f361ce3d723d7dfa2b3cb2ee9a36690427e87add2bc2fc3e74ba82ff95 not found: ID does not exist"
Sep 15 11:52:27.529994 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.529973    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"]
Sep 15 11:52:27.535624 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.535604    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b8fbdd/premium-w4-b8fbdd-kserve-858dc4666-97n5r"]
Sep 15 11:52:27.545837 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.545815    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"]
Sep 15 11:52:27.549216 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.549195    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b8fbdd/distinct-2-w4-b8fbdd-kserve-5f847cc898-rdmd5"]
Sep 15 11:52:27.875246 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.875218    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="616b0cff-38bc-4a86-ac54-e6b83827bea3" path="/var/lib/kubelet/pods/616b0cff-38bc-4a86-ac54-e6b83827bea3/volumes"
Sep 15 11:52:27.875622 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:27.875609    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb2913b3-759c-4dcc-bcc4-fdb3311d1356" path="/var/lib/kubelet/pods/cb2913b3-759c-4dcc-bcc4-fdb3311d1356/volumes"
Sep 15 11:52:31.402153 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.402112    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"]
Sep 15 11:52:31.402612 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.402360    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2" podUID="6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b" containerName="maas-api" containerID="cri-o://242556fe2fc3538a4f184112b43c19c48374c62c12c9e9cb7a9316d9eba75938" gracePeriod=30
Sep 15 11:52:31.458411 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.458381    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm"]
Sep 15 11:52:31.458617 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.458584    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm" podUID="1d8e4f44-3327-49f4-b58a-e2d6d7653e6e" containerName="payload-processing" containerID="cri-o://4abb53605609df859cf0d7cf547db4889e25e1436618a10d996e5dbd7d351325" gracePeriod=30
Sep 15 11:52:31.464978 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.464949    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l"]
Sep 15 11:52:31.465193 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.465170    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l" podUID="b86c579c-5c55-4e23-8a92-2fe23becf514" containerName="payload-pre-processing" containerID="cri-o://a8df4aab47458606e0cc1753794766b4344d61ee64da71718de34fbc9febf7ed" gracePeriod=30
Sep 15 11:52:31.537664 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.537630    2562 generic.go:356] "Generic (PLEG): container finished" podID="6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b" containerID="242556fe2fc3538a4f184112b43c19c48374c62c12c9e9cb7a9316d9eba75938" exitCode=0
Sep 15 11:52:31.537812 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.537697    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2" event={"ID":"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b","Type":"ContainerDied","Data":"242556fe2fc3538a4f184112b43c19c48374c62c12c9e9cb7a9316d9eba75938"}
Sep 15 11:52:31.631946 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.631853    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5a8d6f-djscl"]
Sep 15 11:52:31.633930 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.633887    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5a8d6f-djscl"]
Sep 15 11:52:31.737592 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.737568    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm"
Sep 15 11:52:31.740448 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.740429    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"
Sep 15 11:52:31.750951 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.750930    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l"
Sep 15 11:52:31.813946 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.813919    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxt66\" (UniqueName: \"kubernetes.io/projected/1d8e4f44-3327-49f4-b58a-e2d6d7653e6e-kube-api-access-mxt66\") pod \"1d8e4f44-3327-49f4-b58a-e2d6d7653e6e\" (UID: \"1d8e4f44-3327-49f4-b58a-e2d6d7653e6e\") "
Sep 15 11:52:31.814144 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.813951    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqzcc\" (UniqueName: \"kubernetes.io/projected/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-kube-api-access-mqzcc\") pod \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\" (UID: \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\") "
Sep 15 11:52:31.814144 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.813972    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8e4f44-3327-49f4-b58a-e2d6d7653e6e-plugins-config-volume\") pod \"1d8e4f44-3327-49f4-b58a-e2d6d7653e6e\" (UID: \"1d8e4f44-3327-49f4-b58a-e2d6d7653e6e\") "
Sep 15 11:52:31.814144 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.813990    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-maas-api-tls\") pod \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\" (UID: \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\") "
Sep 15 11:52:31.814144 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.814025    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-openshift-service-ca\") pod \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\" (UID: \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\") "
Sep 15 11:52:31.814144 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.814122    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86c579c-5c55-4e23-8a92-2fe23becf514-plugins-config-volume\") pod \"b86c579c-5c55-4e23-8a92-2fe23becf514\" (UID: \"b86c579c-5c55-4e23-8a92-2fe23becf514\") "
Sep 15 11:52:31.814477 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.814195    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-metrics-certs\") pod \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\" (UID: \"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b\") "
Sep 15 11:52:31.814477 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.814241    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gcw2\" (UniqueName: \"kubernetes.io/projected/b86c579c-5c55-4e23-8a92-2fe23becf514-kube-api-access-7gcw2\") pod \"b86c579c-5c55-4e23-8a92-2fe23becf514\" (UID: \"b86c579c-5c55-4e23-8a92-2fe23becf514\") "
Sep 15 11:52:31.814477 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.814439    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b" (UID: "6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:52:31.814623 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.814472    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d8e4f44-3327-49f4-b58a-e2d6d7653e6e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d8e4f44-3327-49f4-b58a-e2d6d7653e6e" (UID: "1d8e4f44-3327-49f4-b58a-e2d6d7653e6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:52:31.814623 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.814507    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86c579c-5c55-4e23-8a92-2fe23becf514-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b86c579c-5c55-4e23-8a92-2fe23becf514" (UID: "b86c579c-5c55-4e23-8a92-2fe23becf514"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:52:31.814623 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.814568    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-openshift-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:31.816415 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.816381    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b" (UID: "6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:52:31.816534 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.816422    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-kube-api-access-mqzcc" (OuterVolumeSpecName: "kube-api-access-mqzcc") pod "6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b" (UID: "6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b"). InnerVolumeSpecName "kube-api-access-mqzcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:52:31.816534 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.816443    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86c579c-5c55-4e23-8a92-2fe23becf514-kube-api-access-7gcw2" (OuterVolumeSpecName: "kube-api-access-7gcw2") pod "b86c579c-5c55-4e23-8a92-2fe23becf514" (UID: "b86c579c-5c55-4e23-8a92-2fe23becf514"). InnerVolumeSpecName "kube-api-access-7gcw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:52:31.816626 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.816546    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b" (UID: "6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:52:31.816626 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.816608    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d8e4f44-3327-49f4-b58a-e2d6d7653e6e-kube-api-access-mxt66" (OuterVolumeSpecName: "kube-api-access-mxt66") pod "1d8e4f44-3327-49f4-b58a-e2d6d7653e6e" (UID: "1d8e4f44-3327-49f4-b58a-e2d6d7653e6e"). InnerVolumeSpecName "kube-api-access-mxt66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:52:31.874338 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.874312    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0acd70eb-9b86-48d5-88db-479b78bf566e" path="/var/lib/kubelet/pods/0acd70eb-9b86-48d5-88db-479b78bf566e/volumes"
Sep 15 11:52:31.915578 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.915505    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86c579c-5c55-4e23-8a92-2fe23becf514-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:31.915578 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.915528    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-metrics-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:31.915770 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.915611    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gcw2\" (UniqueName: \"kubernetes.io/projected/b86c579c-5c55-4e23-8a92-2fe23becf514-kube-api-access-7gcw2\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:31.915770 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.915707    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxt66\" (UniqueName: \"kubernetes.io/projected/1d8e4f44-3327-49f4-b58a-e2d6d7653e6e-kube-api-access-mxt66\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:31.915770 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.915734    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqzcc\" (UniqueName: \"kubernetes.io/projected/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-kube-api-access-mqzcc\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:31.915770 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.915750    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8e4f44-3327-49f4-b58a-e2d6d7653e6e-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:31.915770 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:31.915764    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b-maas-api-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:32.543507 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.543473    2562 generic.go:356] "Generic (PLEG): container finished" podID="1d8e4f44-3327-49f4-b58a-e2d6d7653e6e" containerID="4abb53605609df859cf0d7cf547db4889e25e1436618a10d996e5dbd7d351325" exitCode=0
Sep 15 11:52:32.543946 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.543521    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm"
Sep 15 11:52:32.543946 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.543563    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm" event={"ID":"1d8e4f44-3327-49f4-b58a-e2d6d7653e6e","Type":"ContainerDied","Data":"4abb53605609df859cf0d7cf547db4889e25e1436618a10d996e5dbd7d351325"}
Sep 15 11:52:32.543946 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.543632    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm" event={"ID":"1d8e4f44-3327-49f4-b58a-e2d6d7653e6e","Type":"ContainerDied","Data":"e4061c400cddd71927679de5dc79b25a4d55930d1e37ba44b57507bb8f9077ae"}
Sep 15 11:52:32.543946 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.543669    2562 scope.go:117] "RemoveContainer" containerID="4abb53605609df859cf0d7cf547db4889e25e1436618a10d996e5dbd7d351325"
Sep 15 11:52:32.545032 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.544940    2562 generic.go:356] "Generic (PLEG): container finished" podID="b86c579c-5c55-4e23-8a92-2fe23becf514" containerID="a8df4aab47458606e0cc1753794766b4344d61ee64da71718de34fbc9febf7ed" exitCode=0
Sep 15 11:52:32.545032 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.544982    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l"
Sep 15 11:52:32.545032 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.544998    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l" event={"ID":"b86c579c-5c55-4e23-8a92-2fe23becf514","Type":"ContainerDied","Data":"a8df4aab47458606e0cc1753794766b4344d61ee64da71718de34fbc9febf7ed"}
Sep 15 11:52:32.545230 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.545045    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l" event={"ID":"b86c579c-5c55-4e23-8a92-2fe23becf514","Type":"ContainerDied","Data":"686b02ca4841edfef1ebdc90f175a2afc6a5e2c0e17aa7700f4c7120dc79b8c5"}
Sep 15 11:52:32.546566 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.546550    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"
Sep 15 11:52:32.546566 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.546554    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2" event={"ID":"6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b","Type":"ContainerDied","Data":"9831291eafaeeba25c3c2043eda33f7b188946b01d63442f63959d65c6e2ca3a"}
Sep 15 11:52:32.553142 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.553122    2562 scope.go:117] "RemoveContainer" containerID="4abb53605609df859cf0d7cf547db4889e25e1436618a10d996e5dbd7d351325"
Sep 15 11:52:32.553384 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:52:32.553360    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4abb53605609df859cf0d7cf547db4889e25e1436618a10d996e5dbd7d351325\": container with ID starting with 4abb53605609df859cf0d7cf547db4889e25e1436618a10d996e5dbd7d351325 not found: ID does not exist" containerID="4abb53605609df859cf0d7cf547db4889e25e1436618a10d996e5dbd7d351325"
Sep 15 11:52:32.553437 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.553392    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4abb53605609df859cf0d7cf547db4889e25e1436618a10d996e5dbd7d351325"} err="failed to get container status \"4abb53605609df859cf0d7cf547db4889e25e1436618a10d996e5dbd7d351325\": rpc error: code = NotFound desc = could not find container \"4abb53605609df859cf0d7cf547db4889e25e1436618a10d996e5dbd7d351325\": container with ID starting with 4abb53605609df859cf0d7cf547db4889e25e1436618a10d996e5dbd7d351325 not found: ID does not exist"
Sep 15 11:52:32.553437 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.553413    2562 scope.go:117] "RemoveContainer" containerID="a8df4aab47458606e0cc1753794766b4344d61ee64da71718de34fbc9febf7ed"
Sep 15 11:52:32.561341 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.561314    2562 scope.go:117] "RemoveContainer" containerID="a8df4aab47458606e0cc1753794766b4344d61ee64da71718de34fbc9febf7ed"
Sep 15 11:52:32.561598 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:52:32.561581    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8df4aab47458606e0cc1753794766b4344d61ee64da71718de34fbc9febf7ed\": container with ID starting with a8df4aab47458606e0cc1753794766b4344d61ee64da71718de34fbc9febf7ed not found: ID does not exist" containerID="a8df4aab47458606e0cc1753794766b4344d61ee64da71718de34fbc9febf7ed"
Sep 15 11:52:32.561679 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.561609    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8df4aab47458606e0cc1753794766b4344d61ee64da71718de34fbc9febf7ed"} err="failed to get container status \"a8df4aab47458606e0cc1753794766b4344d61ee64da71718de34fbc9febf7ed\": rpc error: code = NotFound desc = could not find container \"a8df4aab47458606e0cc1753794766b4344d61ee64da71718de34fbc9febf7ed\": container with ID starting with a8df4aab47458606e0cc1753794766b4344d61ee64da71718de34fbc9febf7ed not found: ID does not exist"
Sep 15 11:52:32.561679 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.561631    2562 scope.go:117] "RemoveContainer" containerID="242556fe2fc3538a4f184112b43c19c48374c62c12c9e9cb7a9316d9eba75938"
Sep 15 11:52:32.583347 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.583311    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm"]
Sep 15 11:52:32.593652 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.593631    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5a8d6f-9dcf78f65-8k6qm"]
Sep 15 11:52:32.608706 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.608680    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l"]
Sep 15 11:52:32.617304 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.613317    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5a8d6f-68955556b6-8n98l"]
Sep 15 11:52:32.628798 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.628773    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"]
Sep 15 11:52:32.634739 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:32.634715    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5a8d6f-6dc66f489b-24qp2"]
Sep 15 11:52:33.872825 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:33.872792    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d8e4f44-3327-49f4-b58a-e2d6d7653e6e" path="/var/lib/kubelet/pods/1d8e4f44-3327-49f4-b58a-e2d6d7653e6e/volumes"
Sep 15 11:52:33.873204 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:33.873177    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b" path="/var/lib/kubelet/pods/6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b/volumes"
Sep 15 11:52:33.873486 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:33.873473    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b86c579c-5c55-4e23-8a92-2fe23becf514" path="/var/lib/kubelet/pods/b86c579c-5c55-4e23-8a92-2fe23becf514/volumes"
Sep 15 11:52:38.396998 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.396967    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d63075-tg2sw"]
Sep 15 11:52:38.397383 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397322    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e09cdb0-f82f-4a5c-a90a-710de63fbacb" containerName="payload-pre-processing"
Sep 15 11:52:38.397383 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397336    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e09cdb0-f82f-4a5c-a90a-710de63fbacb" containerName="payload-pre-processing"
Sep 15 11:52:38.397383 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397344    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d8e4f44-3327-49f4-b58a-e2d6d7653e6e" containerName="payload-processing"
Sep 15 11:52:38.397383 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397349    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d8e4f44-3327-49f4-b58a-e2d6d7653e6e" containerName="payload-processing"
Sep 15 11:52:38.397383 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397363    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb2913b3-759c-4dcc-bcc4-fdb3311d1356" containerName="main"
Sep 15 11:52:38.397383 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397369    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb2913b3-759c-4dcc-bcc4-fdb3311d1356" containerName="main"
Sep 15 11:52:38.397383 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397378    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="616b0cff-38bc-4a86-ac54-e6b83827bea3" containerName="main"
Sep 15 11:52:38.397383 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397383    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="616b0cff-38bc-4a86-ac54-e6b83827bea3" containerName="main"
Sep 15 11:52:38.397662 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397394    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0acd70eb-9b86-48d5-88db-479b78bf566e" containerName="revoke-keys"
Sep 15 11:52:38.397662 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397401    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0acd70eb-9b86-48d5-88db-479b78bf566e" containerName="revoke-keys"
Sep 15 11:52:38.397662 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397409    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b" containerName="maas-api"
Sep 15 11:52:38.397662 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397415    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b" containerName="maas-api"
Sep 15 11:52:38.397662 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397422    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b86c579c-5c55-4e23-8a92-2fe23becf514" containerName="payload-pre-processing"
Sep 15 11:52:38.397662 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397427    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b86c579c-5c55-4e23-8a92-2fe23becf514" containerName="payload-pre-processing"
Sep 15 11:52:38.397662 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397488    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e09cdb0-f82f-4a5c-a90a-710de63fbacb" containerName="payload-pre-processing"
Sep 15 11:52:38.397662 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397497    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb2913b3-759c-4dcc-bcc4-fdb3311d1356" containerName="main"
Sep 15 11:52:38.397662 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397505    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e5899f8-dd4e-4dfb-b3ea-d5d7811bef7b" containerName="maas-api"
Sep 15 11:52:38.397662 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397509    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b86c579c-5c55-4e23-8a92-2fe23becf514" containerName="payload-pre-processing"
Sep 15 11:52:38.397662 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397517    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d8e4f44-3327-49f4-b58a-e2d6d7653e6e" containerName="payload-processing"
Sep 15 11:52:38.397662 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397523    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="616b0cff-38bc-4a86-ac54-e6b83827bea3" containerName="main"
Sep 15 11:52:38.397662 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.397528    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0acd70eb-9b86-48d5-88db-479b78bf566e" containerName="revoke-keys"
Sep 15 11:52:38.401328 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.401309    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d63075-tg2sw"
Sep 15 11:52:38.403260 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.403238    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rbxjk\""
Sep 15 11:52:38.403260 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.403249    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:52:38.408242 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.408198    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d63075-tg2sw"]
Sep 15 11:52:38.468997 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.468972    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77677daa-a0e5-49d6-b136-1b077ad427e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d63075-tg2sw\" (UID: \"77677daa-a0e5-49d6-b136-1b077ad427e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d63075-tg2sw"
Sep 15 11:52:38.569959 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.569935    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77677daa-a0e5-49d6-b136-1b077ad427e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d63075-tg2sw\" (UID: \"77677daa-a0e5-49d6-b136-1b077ad427e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d63075-tg2sw"
Sep 15 11:52:38.570556 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.570539    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77677daa-a0e5-49d6-b136-1b077ad427e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d63075-tg2sw\" (UID: \"77677daa-a0e5-49d6-b136-1b077ad427e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d63075-tg2sw"
Sep 15 11:52:38.713157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.713072    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d63075-tg2sw"
Sep 15 11:52:38.981606 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:38.981534    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d63075-tg2sw"]
Sep 15 11:52:38.984768 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:52:38.984746    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77677daa_a0e5_49d6_b136_1b077ad427e7.slice/crio-941360039c1116b73b4d530cb9fd28f1d710cf9cbe367f3192e9a932a499fc59 WatchSource:0}: Error finding container 941360039c1116b73b4d530cb9fd28f1d710cf9cbe367f3192e9a932a499fc59: Status 404 returned error can't find the container with id 941360039c1116b73b4d530cb9fd28f1d710cf9cbe367f3192e9a932a499fc59
Sep 15 11:52:39.583623 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:39.583587    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d63075-tg2sw" event={"ID":"77677daa-a0e5-49d6-b136-1b077ad427e7","Type":"ContainerStarted","Data":"7fc94dd932844bd4a75698f373f3f641f96e3a570a9c896d98f219ba876cf4b1"}
Sep 15 11:52:39.584024 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:39.583675    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d63075-tg2sw" event={"ID":"77677daa-a0e5-49d6-b136-1b077ad427e7","Type":"ContainerStarted","Data":"941360039c1116b73b4d530cb9fd28f1d710cf9cbe367f3192e9a932a499fc59"}
Sep 15 11:52:39.598443 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:39.598397    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d63075-tg2sw" podStartSLOduration=1.598383248 podStartE2EDuration="1.598383248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:52:39.597655017 +0000 UTC m=+1658.260150209" watchObservedRunningTime="2026-09-15 11:52:39.598383248 +0000 UTC m=+1658.260878429"
Sep 15 11:52:40.590643 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:40.590607    2562 generic.go:356] "Generic (PLEG): container finished" podID="77677daa-a0e5-49d6-b136-1b077ad427e7" containerID="7fc94dd932844bd4a75698f373f3f641f96e3a570a9c896d98f219ba876cf4b1" exitCode=0
Sep 15 11:52:40.591025 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:40.590660    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d63075-tg2sw" event={"ID":"77677daa-a0e5-49d6-b136-1b077ad427e7","Type":"ContainerDied","Data":"7fc94dd932844bd4a75698f373f3f641f96e3a570a9c896d98f219ba876cf4b1"}
Sep 15 11:52:41.719607 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:41.719584    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d63075-tg2sw"
Sep 15 11:52:41.795590 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:41.795562    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77677daa-a0e5-49d6-b136-1b077ad427e7-maas-api-service-ca\") pod \"77677daa-a0e5-49d6-b136-1b077ad427e7\" (UID: \"77677daa-a0e5-49d6-b136-1b077ad427e7\") "
Sep 15 11:52:41.795901 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:41.795879    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77677daa-a0e5-49d6-b136-1b077ad427e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "77677daa-a0e5-49d6-b136-1b077ad427e7" (UID: "77677daa-a0e5-49d6-b136-1b077ad427e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:52:41.897004 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:41.896984    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77677daa-a0e5-49d6-b136-1b077ad427e7-maas-api-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:42.601160 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:42.601135    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d63075-tg2sw"
Sep 15 11:52:42.601338 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:42.601172    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d63075-tg2sw" event={"ID":"77677daa-a0e5-49d6-b136-1b077ad427e7","Type":"ContainerDied","Data":"941360039c1116b73b4d530cb9fd28f1d710cf9cbe367f3192e9a932a499fc59"}
Sep 15 11:52:42.601338 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:42.601220    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="941360039c1116b73b4d530cb9fd28f1d710cf9cbe367f3192e9a932a499fc59"
Sep 15 11:52:43.848616 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:43.848585    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:52:46.328483 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:46.328453    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7459-2a67d0"]
Sep 15 11:52:46.328859 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:46.328809    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77677daa-a0e5-49d6-b136-1b077ad427e7" containerName="revoke-keys"
Sep 15 11:52:46.328859 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:46.328821    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="77677daa-a0e5-49d6-b136-1b077ad427e7" containerName="revoke-keys"
Sep 15 11:52:46.328926 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:46.328889    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="77677daa-a0e5-49d6-b136-1b077ad427e7" containerName="revoke-keys"
Sep 15 11:52:46.331904 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:46.331888    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7459-2a67d0"
Sep 15 11:52:46.334116 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:46.334094    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lfxxr\""
Sep 15 11:52:46.337474 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:46.337452    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7459-2a67d0"]
Sep 15 11:52:46.432724 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:46.432687    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stw6k\" (UniqueName: \"kubernetes.io/projected/7dc5b5f4-594e-481f-af4e-b85f65ba0793-kube-api-access-stw6k\") pod \"test-curl-7459-2a67d0\" (UID: \"7dc5b5f4-594e-481f-af4e-b85f65ba0793\") " pod="opendatahub/test-curl-7459-2a67d0"
Sep 15 11:52:46.534136 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:46.534106    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stw6k\" (UniqueName: \"kubernetes.io/projected/7dc5b5f4-594e-481f-af4e-b85f65ba0793-kube-api-access-stw6k\") pod \"test-curl-7459-2a67d0\" (UID: \"7dc5b5f4-594e-481f-af4e-b85f65ba0793\") " pod="opendatahub/test-curl-7459-2a67d0"
Sep 15 11:52:46.542658 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:46.542641    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stw6k\" (UniqueName: \"kubernetes.io/projected/7dc5b5f4-594e-481f-af4e-b85f65ba0793-kube-api-access-stw6k\") pod \"test-curl-7459-2a67d0\" (UID: \"7dc5b5f4-594e-481f-af4e-b85f65ba0793\") " pod="opendatahub/test-curl-7459-2a67d0"
Sep 15 11:52:46.641114 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:46.641092    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7459-2a67d0"
Sep 15 11:52:46.910093 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:46.909990    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7459-2a67d0"]
Sep 15 11:52:46.912984 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:52:46.912961    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7dc5b5f4_594e_481f_af4e_b85f65ba0793.slice/crio-7a8c7cc87c6a1597a9113d30157ff69324a3cf329fce2fc32adef00f7ac4b3d1 WatchSource:0}: Error finding container 7a8c7cc87c6a1597a9113d30157ff69324a3cf329fce2fc32adef00f7ac4b3d1: Status 404 returned error can't find the container with id 7a8c7cc87c6a1597a9113d30157ff69324a3cf329fce2fc32adef00f7ac4b3d1
Sep 15 11:52:47.626345 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:47.626310    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7459-2a67d0" event={"ID":"7dc5b5f4-594e-481f-af4e-b85f65ba0793","Type":"ContainerStarted","Data":"a2e361b950cae66a3ec7b842d7a0704b820fcd41b001b04b377f338c1b68ce3f"}
Sep 15 11:52:47.626345 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:47.626348    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7459-2a67d0" event={"ID":"7dc5b5f4-594e-481f-af4e-b85f65ba0793","Type":"ContainerStarted","Data":"7a8c7cc87c6a1597a9113d30157ff69324a3cf329fce2fc32adef00f7ac4b3d1"}
Sep 15 11:52:47.641474 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:47.641431    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7459-2a67d0" podStartSLOduration=1.641417572 podStartE2EDuration="1.641417572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:52:47.640929852 +0000 UTC m=+1666.303425059" watchObservedRunningTime="2026-09-15 11:52:47.641417572 +0000 UTC m=+1666.303912753"
Sep 15 11:52:48.268783 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:48.268749    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7459-2a67d0"]
Sep 15 11:52:48.272551 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:48.272529    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7459-2a67d0"]
Sep 15 11:52:48.631477 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:52:48.631444    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7459-2a67d0" container="test-curl-7459-2a67d0"
Sep 15 11:52:48.986189 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:48.986111    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j"]
Sep 15 11:52:48.986390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:48.986364    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j" podUID="8ad53e7e-8857-4115-a455-bee00e78eb6e" containerName="payload-processing" containerID="cri-o://348252064dae31af81102b06caff65af1be9d8e2fb54fd27af663a6a23f83e67" gracePeriod=30
Sep 15 11:52:49.163995 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.163963    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d63075-tg2sw"]
Sep 15 11:52:49.168242 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.168216    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d63075-tg2sw"]
Sep 15 11:52:49.241796 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.241746    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j"
Sep 15 11:52:49.356834 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.356803    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fg7w4\" (UniqueName: \"kubernetes.io/projected/8ad53e7e-8857-4115-a455-bee00e78eb6e-kube-api-access-fg7w4\") pod \"8ad53e7e-8857-4115-a455-bee00e78eb6e\" (UID: \"8ad53e7e-8857-4115-a455-bee00e78eb6e\") "
Sep 15 11:52:49.357009 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.356856    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad53e7e-8857-4115-a455-bee00e78eb6e-plugins-config-volume\") pod \"8ad53e7e-8857-4115-a455-bee00e78eb6e\" (UID: \"8ad53e7e-8857-4115-a455-bee00e78eb6e\") "
Sep 15 11:52:49.357244 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.357221    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad53e7e-8857-4115-a455-bee00e78eb6e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ad53e7e-8857-4115-a455-bee00e78eb6e" (UID: "8ad53e7e-8857-4115-a455-bee00e78eb6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:52:49.358979 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.358952    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad53e7e-8857-4115-a455-bee00e78eb6e-kube-api-access-fg7w4" (OuterVolumeSpecName: "kube-api-access-fg7w4") pod "8ad53e7e-8857-4115-a455-bee00e78eb6e" (UID: "8ad53e7e-8857-4115-a455-bee00e78eb6e"). InnerVolumeSpecName "kube-api-access-fg7w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:52:49.457827 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.457793    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg7w4\" (UniqueName: \"kubernetes.io/projected/8ad53e7e-8857-4115-a455-bee00e78eb6e-kube-api-access-fg7w4\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:49.457827 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.457822    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad53e7e-8857-4115-a455-bee00e78eb6e-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:49.637329 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.637298    2562 generic.go:356] "Generic (PLEG): container finished" podID="8ad53e7e-8857-4115-a455-bee00e78eb6e" containerID="348252064dae31af81102b06caff65af1be9d8e2fb54fd27af663a6a23f83e67" exitCode=0
Sep 15 11:52:49.637726 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.637347    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j"
Sep 15 11:52:49.637726 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.637346    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j" event={"ID":"8ad53e7e-8857-4115-a455-bee00e78eb6e","Type":"ContainerDied","Data":"348252064dae31af81102b06caff65af1be9d8e2fb54fd27af663a6a23f83e67"}
Sep 15 11:52:49.637726 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.637408    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j" event={"ID":"8ad53e7e-8857-4115-a455-bee00e78eb6e","Type":"ContainerDied","Data":"04e194062bb2e19af57c7c7bd28c7e71f7d479f850a9a9fddc5845d6df07331e"}
Sep 15 11:52:49.637726 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.637438    2562 scope.go:117] "RemoveContainer" containerID="348252064dae31af81102b06caff65af1be9d8e2fb54fd27af663a6a23f83e67"
Sep 15 11:52:49.637726 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.637654    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7459-2a67d0" podUID="7dc5b5f4-594e-481f-af4e-b85f65ba0793" containerName="test-curl-7459-2a67d0" containerID="cri-o://a2e361b950cae66a3ec7b842d7a0704b820fcd41b001b04b377f338c1b68ce3f" gracePeriod=2
Sep 15 11:52:49.646248 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.646231    2562 scope.go:117] "RemoveContainer" containerID="348252064dae31af81102b06caff65af1be9d8e2fb54fd27af663a6a23f83e67"
Sep 15 11:52:49.646502 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:52:49.646485    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348252064dae31af81102b06caff65af1be9d8e2fb54fd27af663a6a23f83e67\": container with ID starting with 348252064dae31af81102b06caff65af1be9d8e2fb54fd27af663a6a23f83e67 not found: ID does not exist" containerID="348252064dae31af81102b06caff65af1be9d8e2fb54fd27af663a6a23f83e67"
Sep 15 11:52:49.646546 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.646511    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348252064dae31af81102b06caff65af1be9d8e2fb54fd27af663a6a23f83e67"} err="failed to get container status \"348252064dae31af81102b06caff65af1be9d8e2fb54fd27af663a6a23f83e67\": rpc error: code = NotFound desc = could not find container \"348252064dae31af81102b06caff65af1be9d8e2fb54fd27af663a6a23f83e67\": container with ID starting with 348252064dae31af81102b06caff65af1be9d8e2fb54fd27af663a6a23f83e67 not found: ID does not exist"
Sep 15 11:52:49.647991 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:52:49.647970    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7459-2a67d0\" is forbidden: User \"system:node:ip-10-0-135-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-143.ec2.internal' and this object" podUID="7dc5b5f4-594e-481f-af4e-b85f65ba0793" pod="opendatahub/test-curl-7459-2a67d0"
Sep 15 11:52:49.672784 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.672762    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j"]
Sep 15 11:52:49.675751 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:52:49.675719    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7459-2a67d0\" is forbidden: User \"system:node:ip-10-0-135-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-143.ec2.internal' and this object" podUID="7dc5b5f4-594e-481f-af4e-b85f65ba0793" pod="opendatahub/test-curl-7459-2a67d0"
Sep 15 11:52:49.675853 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.675755    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d63075-768974c9d5-xzk4j"]
Sep 15 11:52:49.873075 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.873026    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77677daa-a0e5-49d6-b136-1b077ad427e7" path="/var/lib/kubelet/pods/77677daa-a0e5-49d6-b136-1b077ad427e7/volumes"
Sep 15 11:52:49.873492 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:49.873470    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ad53e7e-8857-4115-a455-bee00e78eb6e" path="/var/lib/kubelet/pods/8ad53e7e-8857-4115-a455-bee00e78eb6e/volumes"
Sep 15 11:52:51.809695 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:51.809673    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7459-2a67d0"
Sep 15 11:52:51.878916 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:51.878888    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stw6k\" (UniqueName: \"kubernetes.io/projected/7dc5b5f4-594e-481f-af4e-b85f65ba0793-kube-api-access-stw6k\") pod \"7dc5b5f4-594e-481f-af4e-b85f65ba0793\" (UID: \"7dc5b5f4-594e-481f-af4e-b85f65ba0793\") "
Sep 15 11:52:51.880862 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:51.880840    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc5b5f4-594e-481f-af4e-b85f65ba0793-kube-api-access-stw6k" (OuterVolumeSpecName: "kube-api-access-stw6k") pod "7dc5b5f4-594e-481f-af4e-b85f65ba0793" (UID: "7dc5b5f4-594e-481f-af4e-b85f65ba0793"). InnerVolumeSpecName "kube-api-access-stw6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:52:51.980048 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:51.979986    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stw6k\" (UniqueName: \"kubernetes.io/projected/7dc5b5f4-594e-481f-af4e-b85f65ba0793-kube-api-access-stw6k\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:52:52.654106 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:52.654072    2562 generic.go:356] "Generic (PLEG): container finished" podID="7dc5b5f4-594e-481f-af4e-b85f65ba0793" containerID="a2e361b950cae66a3ec7b842d7a0704b820fcd41b001b04b377f338c1b68ce3f" exitCode=137
Sep 15 11:52:52.654106 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:52.654108    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7459-2a67d0"
Sep 15 11:52:52.654344 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:52.654142    2562 scope.go:117] "RemoveContainer" containerID="a2e361b950cae66a3ec7b842d7a0704b820fcd41b001b04b377f338c1b68ce3f"
Sep 15 11:52:52.662737 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:52.662720    2562 scope.go:117] "RemoveContainer" containerID="a2e361b950cae66a3ec7b842d7a0704b820fcd41b001b04b377f338c1b68ce3f"
Sep 15 11:52:52.663015 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:52:52.662998    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2e361b950cae66a3ec7b842d7a0704b820fcd41b001b04b377f338c1b68ce3f\": container with ID starting with a2e361b950cae66a3ec7b842d7a0704b820fcd41b001b04b377f338c1b68ce3f not found: ID does not exist" containerID="a2e361b950cae66a3ec7b842d7a0704b820fcd41b001b04b377f338c1b68ce3f"
Sep 15 11:52:52.663103 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:52.663021    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2e361b950cae66a3ec7b842d7a0704b820fcd41b001b04b377f338c1b68ce3f"} err="failed to get container status \"a2e361b950cae66a3ec7b842d7a0704b820fcd41b001b04b377f338c1b68ce3f\": rpc error: code = NotFound desc = could not find container \"a2e361b950cae66a3ec7b842d7a0704b820fcd41b001b04b377f338c1b68ce3f\": container with ID starting with a2e361b950cae66a3ec7b842d7a0704b820fcd41b001b04b377f338c1b68ce3f not found: ID does not exist"
Sep 15 11:52:53.873159 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:53.873122    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dc5b5f4-594e-481f-af4e-b85f65ba0793" path="/var/lib/kubelet/pods/7dc5b5f4-594e-481f-af4e-b85f65ba0793/volumes"
Sep 15 11:52:55.357505 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:52:55.357471    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:53:11.843784 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:53:11.843752    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:53:36.550409 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:53:36.550328    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:54:02.753932 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:02.753895    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:54:17.741837 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:17.741791    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:54:26.110437 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:26.110403    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7459-730649"]
Sep 15 11:54:26.110805 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:26.110726    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ad53e7e-8857-4115-a455-bee00e78eb6e" containerName="payload-processing"
Sep 15 11:54:26.110805 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:26.110737    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ad53e7e-8857-4115-a455-bee00e78eb6e" containerName="payload-processing"
Sep 15 11:54:26.110805 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:26.110750    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dc5b5f4-594e-481f-af4e-b85f65ba0793" containerName="test-curl-7459-2a67d0"
Sep 15 11:54:26.110805 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:26.110756    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dc5b5f4-594e-481f-af4e-b85f65ba0793" containerName="test-curl-7459-2a67d0"
Sep 15 11:54:26.111124 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:26.110807    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ad53e7e-8857-4115-a455-bee00e78eb6e" containerName="payload-processing"
Sep 15 11:54:26.111124 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:26.110816    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dc5b5f4-594e-481f-af4e-b85f65ba0793" containerName="test-curl-7459-2a67d0"
Sep 15 11:54:26.113508 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:26.113491    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7459-730649"
Sep 15 11:54:26.115576 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:26.115554    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lfxxr\""
Sep 15 11:54:26.118704 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:26.118681    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7459-730649"]
Sep 15 11:54:26.175166 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:26.175138    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8pfm\" (UniqueName: \"kubernetes.io/projected/014054d7-6f39-428d-a8f3-5cb9aa595ae1-kube-api-access-v8pfm\") pod \"test-curl-7459-730649\" (UID: \"014054d7-6f39-428d-a8f3-5cb9aa595ae1\") " pod="opendatahub/test-curl-7459-730649"
Sep 15 11:54:26.275715 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:26.275688    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8pfm\" (UniqueName: \"kubernetes.io/projected/014054d7-6f39-428d-a8f3-5cb9aa595ae1-kube-api-access-v8pfm\") pod \"test-curl-7459-730649\" (UID: \"014054d7-6f39-428d-a8f3-5cb9aa595ae1\") " pod="opendatahub/test-curl-7459-730649"
Sep 15 11:54:26.284524 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:26.284494    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8pfm\" (UniqueName: \"kubernetes.io/projected/014054d7-6f39-428d-a8f3-5cb9aa595ae1-kube-api-access-v8pfm\") pod \"test-curl-7459-730649\" (UID: \"014054d7-6f39-428d-a8f3-5cb9aa595ae1\") " pod="opendatahub/test-curl-7459-730649"
Sep 15 11:54:26.423284 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:26.423198    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7459-730649"
Sep 15 11:54:26.700126 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:26.700040    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7459-730649"]
Sep 15 11:54:26.705573 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:54:26.705545    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod014054d7_6f39_428d_a8f3_5cb9aa595ae1.slice/crio-fd31bd769602dcdedfb38983e5fd13be9c4e983879a8bf6c7181e3b29d1ffe5e WatchSource:0}: Error finding container fd31bd769602dcdedfb38983e5fd13be9c4e983879a8bf6c7181e3b29d1ffe5e: Status 404 returned error can't find the container with id fd31bd769602dcdedfb38983e5fd13be9c4e983879a8bf6c7181e3b29d1ffe5e
Sep 15 11:54:27.072730 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:27.072630    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7459-730649" event={"ID":"014054d7-6f39-428d-a8f3-5cb9aa595ae1","Type":"ContainerStarted","Data":"9a90574dfe77fc0b47fa033f3befd244e3f5ab36859ba653fe0cad3f465fd6f6"}
Sep 15 11:54:27.072730 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:27.072671    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7459-730649" event={"ID":"014054d7-6f39-428d-a8f3-5cb9aa595ae1","Type":"ContainerStarted","Data":"fd31bd769602dcdedfb38983e5fd13be9c4e983879a8bf6c7181e3b29d1ffe5e"}
Sep 15 11:54:27.689978 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:27.689921    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7459-730649" podStartSLOduration=1.6899038210000001 podStartE2EDuration="1.689903821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:54:27.085575018 +0000 UTC m=+1765.748070198" watchObservedRunningTime="2026-09-15 11:54:27.689903821 +0000 UTC m=+1766.352399001"
Sep 15 11:54:27.691558 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:27.691528    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7459-730649"]
Sep 15 11:54:27.694164 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:27.694134    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7459-730649"]
Sep 15 11:54:27.807010 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:27.806929    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7459-9d490e"]
Sep 15 11:54:27.807291 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:27.807278    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="014054d7-6f39-428d-a8f3-5cb9aa595ae1" containerName="test-curl-7459-730649"
Sep 15 11:54:27.807353 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:27.807292    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="014054d7-6f39-428d-a8f3-5cb9aa595ae1" containerName="test-curl-7459-730649"
Sep 15 11:54:27.807392 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:27.807371    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="014054d7-6f39-428d-a8f3-5cb9aa595ae1" containerName="test-curl-7459-730649"
Sep 15 11:54:27.810185 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:27.810168    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7459-9d490e"
Sep 15 11:54:27.817444 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:27.817115    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7459-9d490e"]
Sep 15 11:54:27.890957 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:27.890926    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7mnn\" (UniqueName: \"kubernetes.io/projected/e17ae297-ebee-46e4-a2ff-8a9511532ab0-kube-api-access-s7mnn\") pod \"test-curl-7459-9d490e\" (UID: \"e17ae297-ebee-46e4-a2ff-8a9511532ab0\") " pod="opendatahub/test-curl-7459-9d490e"
Sep 15 11:54:27.991974 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:27.991937    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7mnn\" (UniqueName: \"kubernetes.io/projected/e17ae297-ebee-46e4-a2ff-8a9511532ab0-kube-api-access-s7mnn\") pod \"test-curl-7459-9d490e\" (UID: \"e17ae297-ebee-46e4-a2ff-8a9511532ab0\") " pod="opendatahub/test-curl-7459-9d490e"
Sep 15 11:54:27.999432 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:27.999410    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7mnn\" (UniqueName: \"kubernetes.io/projected/e17ae297-ebee-46e4-a2ff-8a9511532ab0-kube-api-access-s7mnn\") pod \"test-curl-7459-9d490e\" (UID: \"e17ae297-ebee-46e4-a2ff-8a9511532ab0\") " pod="opendatahub/test-curl-7459-9d490e"
Sep 15 11:54:28.077830 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:54:28.077761    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7459-730649" container="test-curl-7459-730649"
Sep 15 11:54:28.120131 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:28.120088    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7459-9d490e"
Sep 15 11:54:28.385854 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:28.385826    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7459-9d490e"]
Sep 15 11:54:28.388358 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:54:28.388329    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode17ae297_ebee_46e4_a2ff_8a9511532ab0.slice/crio-f9df540ce27a3c132720be8254a2415ee3f4d153f733eeea104849277796cb57 WatchSource:0}: Error finding container f9df540ce27a3c132720be8254a2415ee3f4d153f733eeea104849277796cb57: Status 404 returned error can't find the container with id f9df540ce27a3c132720be8254a2415ee3f4d153f733eeea104849277796cb57
Sep 15 11:54:29.084124 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:29.084085    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7459-9d490e" event={"ID":"e17ae297-ebee-46e4-a2ff-8a9511532ab0","Type":"ContainerStarted","Data":"519b12541eea52f0ccfc3bbfd67f29ea99db373b3cb913b1ca3d7904c8b6ce1c"}
Sep 15 11:54:29.084609 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:29.084135    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7459-9d490e" event={"ID":"e17ae297-ebee-46e4-a2ff-8a9511532ab0","Type":"ContainerStarted","Data":"f9df540ce27a3c132720be8254a2415ee3f4d153f733eeea104849277796cb57"}
Sep 15 11:54:29.084609 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:29.084182    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7459-730649" podUID="014054d7-6f39-428d-a8f3-5cb9aa595ae1" containerName="test-curl-7459-730649" containerID="cri-o://9a90574dfe77fc0b47fa033f3befd244e3f5ab36859ba653fe0cad3f465fd6f6" gracePeriod=2
Sep 15 11:54:29.099732 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:29.099686    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7459-9d490e" podStartSLOduration=2.09967237 podStartE2EDuration="2.09967237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:54:29.098189568 +0000 UTC m=+1767.760684761" watchObservedRunningTime="2026-09-15 11:54:29.09967237 +0000 UTC m=+1767.762167552"
Sep 15 11:54:29.679207 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:29.679170    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7459-9d490e"]
Sep 15 11:54:29.683110 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:29.683085    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7459-9d490e"]
Sep 15 11:54:30.088534 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:54:30.088495    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7459-9d490e" container="test-curl-7459-9d490e"
Sep 15 11:54:31.093273 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:31.093240    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7459-9d490e" podUID="e17ae297-ebee-46e4-a2ff-8a9511532ab0" containerName="test-curl-7459-9d490e" containerID="cri-o://519b12541eea52f0ccfc3bbfd67f29ea99db373b3cb913b1ca3d7904c8b6ce1c" gracePeriod=2
Sep 15 11:54:31.154756 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:31.154681    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:54:31.236222 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:31.236195    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7459-730649"
Sep 15 11:54:31.321160 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:31.321135    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8pfm\" (UniqueName: \"kubernetes.io/projected/014054d7-6f39-428d-a8f3-5cb9aa595ae1-kube-api-access-v8pfm\") pod \"014054d7-6f39-428d-a8f3-5cb9aa595ae1\" (UID: \"014054d7-6f39-428d-a8f3-5cb9aa595ae1\") "
Sep 15 11:54:31.323098 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:31.323071    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014054d7-6f39-428d-a8f3-5cb9aa595ae1-kube-api-access-v8pfm" (OuterVolumeSpecName: "kube-api-access-v8pfm") pod "014054d7-6f39-428d-a8f3-5cb9aa595ae1" (UID: "014054d7-6f39-428d-a8f3-5cb9aa595ae1"). InnerVolumeSpecName "kube-api-access-v8pfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:54:31.422739 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:31.422667    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8pfm\" (UniqueName: \"kubernetes.io/projected/014054d7-6f39-428d-a8f3-5cb9aa595ae1-kube-api-access-v8pfm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:54:31.872867 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:31.872835    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="014054d7-6f39-428d-a8f3-5cb9aa595ae1" path="/var/lib/kubelet/pods/014054d7-6f39-428d-a8f3-5cb9aa595ae1/volumes"
Sep 15 11:54:32.099000 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:32.098966    2562 generic.go:356] "Generic (PLEG): container finished" podID="014054d7-6f39-428d-a8f3-5cb9aa595ae1" containerID="9a90574dfe77fc0b47fa033f3befd244e3f5ab36859ba653fe0cad3f465fd6f6" exitCode=137
Sep 15 11:54:32.099445 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:32.099013    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7459-730649"
Sep 15 11:54:32.099445 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:32.099074    2562 scope.go:117] "RemoveContainer" containerID="9a90574dfe77fc0b47fa033f3befd244e3f5ab36859ba653fe0cad3f465fd6f6"
Sep 15 11:54:32.108144 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:32.108045    2562 scope.go:117] "RemoveContainer" containerID="9a90574dfe77fc0b47fa033f3befd244e3f5ab36859ba653fe0cad3f465fd6f6"
Sep 15 11:54:32.109428 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:54:32.109404    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a90574dfe77fc0b47fa033f3befd244e3f5ab36859ba653fe0cad3f465fd6f6\": container with ID starting with 9a90574dfe77fc0b47fa033f3befd244e3f5ab36859ba653fe0cad3f465fd6f6 not found: ID does not exist" containerID="9a90574dfe77fc0b47fa033f3befd244e3f5ab36859ba653fe0cad3f465fd6f6"
Sep 15 11:54:32.109542 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:32.109434    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a90574dfe77fc0b47fa033f3befd244e3f5ab36859ba653fe0cad3f465fd6f6"} err="failed to get container status \"9a90574dfe77fc0b47fa033f3befd244e3f5ab36859ba653fe0cad3f465fd6f6\": rpc error: code = NotFound desc = could not find container \"9a90574dfe77fc0b47fa033f3befd244e3f5ab36859ba653fe0cad3f465fd6f6\": container with ID starting with 9a90574dfe77fc0b47fa033f3befd244e3f5ab36859ba653fe0cad3f465fd6f6 not found: ID does not exist"
Sep 15 11:54:33.244619 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:33.244591    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7459-9d490e"
Sep 15 11:54:33.340198 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:33.340164    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7mnn\" (UniqueName: \"kubernetes.io/projected/e17ae297-ebee-46e4-a2ff-8a9511532ab0-kube-api-access-s7mnn\") pod \"e17ae297-ebee-46e4-a2ff-8a9511532ab0\" (UID: \"e17ae297-ebee-46e4-a2ff-8a9511532ab0\") "
Sep 15 11:54:33.342204 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:33.342179    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17ae297-ebee-46e4-a2ff-8a9511532ab0-kube-api-access-s7mnn" (OuterVolumeSpecName: "kube-api-access-s7mnn") pod "e17ae297-ebee-46e4-a2ff-8a9511532ab0" (UID: "e17ae297-ebee-46e4-a2ff-8a9511532ab0"). InnerVolumeSpecName "kube-api-access-s7mnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:54:33.441237 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:33.441207    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7mnn\" (UniqueName: \"kubernetes.io/projected/e17ae297-ebee-46e4-a2ff-8a9511532ab0-kube-api-access-s7mnn\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:54:33.873118 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:33.873085    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e17ae297-ebee-46e4-a2ff-8a9511532ab0" path="/var/lib/kubelet/pods/e17ae297-ebee-46e4-a2ff-8a9511532ab0/volumes"
Sep 15 11:54:34.115998 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:34.115952    2562 generic.go:356] "Generic (PLEG): container finished" podID="e17ae297-ebee-46e4-a2ff-8a9511532ab0" containerID="519b12541eea52f0ccfc3bbfd67f29ea99db373b3cb913b1ca3d7904c8b6ce1c" exitCode=137
Sep 15 11:54:34.116312 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:34.116004    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7459-9d490e"
Sep 15 11:54:34.116312 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:34.116020    2562 scope.go:117] "RemoveContainer" containerID="519b12541eea52f0ccfc3bbfd67f29ea99db373b3cb913b1ca3d7904c8b6ce1c"
Sep 15 11:54:34.124248 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:34.124227    2562 scope.go:117] "RemoveContainer" containerID="519b12541eea52f0ccfc3bbfd67f29ea99db373b3cb913b1ca3d7904c8b6ce1c"
Sep 15 11:54:34.124565 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:54:34.124545    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"519b12541eea52f0ccfc3bbfd67f29ea99db373b3cb913b1ca3d7904c8b6ce1c\": container with ID starting with 519b12541eea52f0ccfc3bbfd67f29ea99db373b3cb913b1ca3d7904c8b6ce1c not found: ID does not exist" containerID="519b12541eea52f0ccfc3bbfd67f29ea99db373b3cb913b1ca3d7904c8b6ce1c"
Sep 15 11:54:34.124613 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:34.124576    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519b12541eea52f0ccfc3bbfd67f29ea99db373b3cb913b1ca3d7904c8b6ce1c"} err="failed to get container status \"519b12541eea52f0ccfc3bbfd67f29ea99db373b3cb913b1ca3d7904c8b6ce1c\": rpc error: code = NotFound desc = could not find container \"519b12541eea52f0ccfc3bbfd67f29ea99db373b3cb913b1ca3d7904c8b6ce1c\": container with ID starting with 519b12541eea52f0ccfc3bbfd67f29ea99db373b3cb913b1ca3d7904c8b6ce1c not found: ID does not exist"
Sep 15 11:54:38.154608 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.154575    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"]
Sep 15 11:54:38.155083 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.155049    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e17ae297-ebee-46e4-a2ff-8a9511532ab0" containerName="test-curl-7459-9d490e"
Sep 15 11:54:38.155138 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.155089    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e17ae297-ebee-46e4-a2ff-8a9511532ab0" containerName="test-curl-7459-9d490e"
Sep 15 11:54:38.155193 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.155182    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e17ae297-ebee-46e4-a2ff-8a9511532ab0" containerName="test-curl-7459-9d490e"
Sep 15 11:54:38.164629 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.164608    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:38.167761 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.167736    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7efa8b\"/\"kube-root-ca.crt\""
Sep 15 11:54:38.167894 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.167770    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7efa8b\"/\"default-dockercfg-l8jb7\""
Sep 15 11:54:38.167969 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.167830    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7efa8b\"/\"rate-test-model-7efa8b-kserve-self-signed-certs\""
Sep 15 11:54:38.167969 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.167848    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7efa8b\"/\"openshift-service-ca.crt\""
Sep 15 11:54:38.168868 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.168844    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"]
Sep 15 11:54:38.282048 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.282018    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-dshm\") pod \"rate-test-model-7efa8b-kserve-686b49df99-kp92g\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") " pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:38.282208 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.282048    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfk4n\" (UniqueName: \"kubernetes.io/projected/d828888d-93ab-4e90-81cb-3127f7646279-kube-api-access-hfk4n\") pod \"rate-test-model-7efa8b-kserve-686b49df99-kp92g\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") " pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:38.282208 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.282094    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-home\") pod \"rate-test-model-7efa8b-kserve-686b49df99-kp92g\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") " pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:38.282208 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.282111    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-tmp-dir\") pod \"rate-test-model-7efa8b-kserve-686b49df99-kp92g\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") " pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:38.282208 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.282148    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d828888d-93ab-4e90-81cb-3127f7646279-tls-certs\") pod \"rate-test-model-7efa8b-kserve-686b49df99-kp92g\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") " pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:38.282208 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.282167    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-model-cache\") pod \"rate-test-model-7efa8b-kserve-686b49df99-kp92g\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") " pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:38.383107 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.383050    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-dshm\") pod \"rate-test-model-7efa8b-kserve-686b49df99-kp92g\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") " pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:38.383251 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.383115    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfk4n\" (UniqueName: \"kubernetes.io/projected/d828888d-93ab-4e90-81cb-3127f7646279-kube-api-access-hfk4n\") pod \"rate-test-model-7efa8b-kserve-686b49df99-kp92g\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") " pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:38.383251 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.383158    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-home\") pod \"rate-test-model-7efa8b-kserve-686b49df99-kp92g\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") " pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:38.383251 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.383182    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-tmp-dir\") pod \"rate-test-model-7efa8b-kserve-686b49df99-kp92g\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") " pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:38.383251 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.383217    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d828888d-93ab-4e90-81cb-3127f7646279-tls-certs\") pod \"rate-test-model-7efa8b-kserve-686b49df99-kp92g\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") " pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:38.383251 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.383245    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-model-cache\") pod \"rate-test-model-7efa8b-kserve-686b49df99-kp92g\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") " pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:38.383562 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.383541    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-home\") pod \"rate-test-model-7efa8b-kserve-686b49df99-kp92g\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") " pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:38.383562 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.383556    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-tmp-dir\") pod \"rate-test-model-7efa8b-kserve-686b49df99-kp92g\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") " pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:38.383678 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.383597    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-model-cache\") pod \"rate-test-model-7efa8b-kserve-686b49df99-kp92g\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") " pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:38.385263 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.385248    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-dshm\") pod \"rate-test-model-7efa8b-kserve-686b49df99-kp92g\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") " pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:38.385692 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.385673    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d828888d-93ab-4e90-81cb-3127f7646279-tls-certs\") pod \"rate-test-model-7efa8b-kserve-686b49df99-kp92g\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") " pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:38.392807 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.392788    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfk4n\" (UniqueName: \"kubernetes.io/projected/d828888d-93ab-4e90-81cb-3127f7646279-kube-api-access-hfk4n\") pod \"rate-test-model-7efa8b-kserve-686b49df99-kp92g\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") " pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:38.475213 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.475158    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:38.744910 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:38.744882    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"]
Sep 15 11:54:38.747561 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:54:38.747538    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd828888d_93ab_4e90_81cb_3127f7646279.slice/crio-e447373a1df7de3f28335ec8431fcbc619b79621af7f635469454dc2074c9e37 WatchSource:0}: Error finding container e447373a1df7de3f28335ec8431fcbc619b79621af7f635469454dc2074c9e37: Status 404 returned error can't find the container with id e447373a1df7de3f28335ec8431fcbc619b79621af7f635469454dc2074c9e37
Sep 15 11:54:39.140591 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:39.140552    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g" event={"ID":"d828888d-93ab-4e90-81cb-3127f7646279","Type":"ContainerStarted","Data":"703d153a318d43bc33e8e10e31278decd59d3737a40c6a9b3a7191074db2ab64"}
Sep 15 11:54:39.140735 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:39.140599    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g" event={"ID":"d828888d-93ab-4e90-81cb-3127f7646279","Type":"ContainerStarted","Data":"e447373a1df7de3f28335ec8431fcbc619b79621af7f635469454dc2074c9e37"}
Sep 15 11:54:39.159812 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:39.159753    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g" podStartSLOduration=1.159732833 podStartE2EDuration="1.159732833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:54:39.158681102 +0000 UTC m=+1777.821176300" watchObservedRunningTime="2026-09-15 11:54:39.159732833 +0000 UTC m=+1777.822228014"
Sep 15 11:54:40.758272 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:40.758240    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:54:48.475384 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:48.475349    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:48.475792 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:48.475401    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:48.488086 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:48.488047    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:54:49.196400 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:54:49.196369    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:55:12.649867 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:12.649830    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:55:25.566139 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:25.566101    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:55:34.789226 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:34.789188    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:55:39.580385 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:39.580348    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"]
Sep 15 11:55:45.765660 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:45.765623    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:55:47.579008 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.578957    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2"]
Sep 15 11:55:47.582190 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.582173    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq"]
Sep 15 11:55:47.582306 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.582288    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2"
Sep 15 11:55:47.584191 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.584170    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-173324\""
Sep 15 11:55:47.584808 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.584792    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-173324-dockercfg-knshx\""
Sep 15 11:55:47.585188 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.585174    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq"
Sep 15 11:55:47.593382 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.593358    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2"]
Sep 15 11:55:47.596756 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.596733    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq"]
Sep 15 11:55:47.675538 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.675503    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-648l9\" (UniqueName: \"kubernetes.io/projected/c1def9c7-31a0-4398-883d-db966eb6741e-kube-api-access-648l9\") pod \"payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq\" (UID: \"c1def9c7-31a0-4398-883d-db966eb6741e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq"
Sep 15 11:55:47.675713 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.675547    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlqfl\" (UniqueName: \"kubernetes.io/projected/3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f-kube-api-access-jlqfl\") pod \"payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2\" (UID: \"3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2"
Sep 15 11:55:47.675713 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.675623    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1def9c7-31a0-4398-883d-db966eb6741e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq\" (UID: \"c1def9c7-31a0-4398-883d-db966eb6741e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq"
Sep 15 11:55:47.675713 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.675678    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2\" (UID: \"3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2"
Sep 15 11:55:47.776355 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.776323    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1def9c7-31a0-4398-883d-db966eb6741e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq\" (UID: \"c1def9c7-31a0-4398-883d-db966eb6741e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq"
Sep 15 11:55:47.776561 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.776363    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2\" (UID: \"3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2"
Sep 15 11:55:47.776561 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.776396    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-648l9\" (UniqueName: \"kubernetes.io/projected/c1def9c7-31a0-4398-883d-db966eb6741e-kube-api-access-648l9\") pod \"payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq\" (UID: \"c1def9c7-31a0-4398-883d-db966eb6741e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq"
Sep 15 11:55:47.776561 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.776431    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlqfl\" (UniqueName: \"kubernetes.io/projected/3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f-kube-api-access-jlqfl\") pod \"payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2\" (UID: \"3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2"
Sep 15 11:55:47.777127 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.777079    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1def9c7-31a0-4398-883d-db966eb6741e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq\" (UID: \"c1def9c7-31a0-4398-883d-db966eb6741e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq"
Sep 15 11:55:47.777227 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.777129    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2\" (UID: \"3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2"
Sep 15 11:55:47.784032 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.784012    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlqfl\" (UniqueName: \"kubernetes.io/projected/3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f-kube-api-access-jlqfl\") pod \"payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2\" (UID: \"3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2"
Sep 15 11:55:47.784139 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.784051    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-648l9\" (UniqueName: \"kubernetes.io/projected/c1def9c7-31a0-4398-883d-db966eb6741e-kube-api-access-648l9\") pod \"payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq\" (UID: \"c1def9c7-31a0-4398-883d-db966eb6741e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq"
Sep 15 11:55:47.894493 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.894466    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2"
Sep 15 11:55:47.901493 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:47.901475    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq"
Sep 15 11:55:48.170803 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:48.170775    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2"]
Sep 15 11:55:48.173337 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:55:48.173308    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3058e8c1_f6c4_4c7f_b2d1_dbb5f098a10f.slice/crio-8a3e60e1a2f68c98f068f152075f1a0698f2f605365f197956a879ce3f687026 WatchSource:0}: Error finding container 8a3e60e1a2f68c98f068f152075f1a0698f2f605365f197956a879ce3f687026: Status 404 returned error can't find the container with id 8a3e60e1a2f68c98f068f152075f1a0698f2f605365f197956a879ce3f687026
Sep 15 11:55:48.186946 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:48.186919    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq"]
Sep 15 11:55:48.189359 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:55:48.189336    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1def9c7_31a0_4398_883d_db966eb6741e.slice/crio-cb3dd4d40ac4c3ed8515485e0199c505052d622f69283df8e7b9d382698bc19b WatchSource:0}: Error finding container cb3dd4d40ac4c3ed8515485e0199c505052d622f69283df8e7b9d382698bc19b: Status 404 returned error can't find the container with id cb3dd4d40ac4c3ed8515485e0199c505052d622f69283df8e7b9d382698bc19b
Sep 15 11:55:48.456972 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:48.456862    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2" event={"ID":"3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f","Type":"ContainerStarted","Data":"ebe6c452f0ff220571610897a16501711f55a8e570d1889fdc6916590ed5136e"}
Sep 15 11:55:48.456972 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:48.456927    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2" event={"ID":"3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f","Type":"ContainerStarted","Data":"8a3e60e1a2f68c98f068f152075f1a0698f2f605365f197956a879ce3f687026"}
Sep 15 11:55:48.456972 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:48.456954    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2"
Sep 15 11:55:48.458601 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:48.458576    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq" event={"ID":"c1def9c7-31a0-4398-883d-db966eb6741e","Type":"ContainerStarted","Data":"a1657f6186404f5c70a47a5400c20dd3182da77e473854441b6b0a002e7ab824"}
Sep 15 11:55:48.458727 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:48.458621    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq" event={"ID":"c1def9c7-31a0-4398-883d-db966eb6741e","Type":"ContainerStarted","Data":"cb3dd4d40ac4c3ed8515485e0199c505052d622f69283df8e7b9d382698bc19b"}
Sep 15 11:55:48.458727 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:48.458701    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq"
Sep 15 11:55:48.476783 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:48.476716    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2" podStartSLOduration=1.476701263 podStartE2EDuration="1.476701263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:55:48.475632832 +0000 UTC m=+1847.138128013" watchObservedRunningTime="2026-09-15 11:55:48.476701263 +0000 UTC m=+1847.139196445"
Sep 15 11:55:48.493724 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:48.493674    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq" podStartSLOduration=1.493661693 podStartE2EDuration="1.493661693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:55:48.492524952 +0000 UTC m=+1847.155020155" watchObservedRunningTime="2026-09-15 11:55:48.493661693 +0000 UTC m=+1847.156156873"
Sep 15 11:55:54.598922 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:54.598886    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g" podUID="d828888d-93ab-4e90-81cb-3127f7646279" containerName="main" containerID="cri-o://703d153a318d43bc33e8e10e31278decd59d3737a40c6a9b3a7191074db2ab64" gracePeriod=45
Sep 15 11:55:54.838390 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:54.838365    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:55:54.934345 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:54.934273    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d828888d-93ab-4e90-81cb-3127f7646279-tls-certs\") pod \"d828888d-93ab-4e90-81cb-3127f7646279\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") "
Sep 15 11:55:54.934345 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:54.934329    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-home\") pod \"d828888d-93ab-4e90-81cb-3127f7646279\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") "
Sep 15 11:55:54.934561 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:54.934355    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfk4n\" (UniqueName: \"kubernetes.io/projected/d828888d-93ab-4e90-81cb-3127f7646279-kube-api-access-hfk4n\") pod \"d828888d-93ab-4e90-81cb-3127f7646279\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") "
Sep 15 11:55:54.934561 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:54.934447    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-model-cache\") pod \"d828888d-93ab-4e90-81cb-3127f7646279\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") "
Sep 15 11:55:54.934561 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:54.934486    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-dshm\") pod \"d828888d-93ab-4e90-81cb-3127f7646279\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") "
Sep 15 11:55:54.934561 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:54.934524    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-tmp-dir\") pod \"d828888d-93ab-4e90-81cb-3127f7646279\" (UID: \"d828888d-93ab-4e90-81cb-3127f7646279\") "
Sep 15 11:55:54.934773 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:54.934571    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-home" (OuterVolumeSpecName: "home") pod "d828888d-93ab-4e90-81cb-3127f7646279" (UID: "d828888d-93ab-4e90-81cb-3127f7646279"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:55:54.934773 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:54.934688    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-model-cache" (OuterVolumeSpecName: "model-cache") pod "d828888d-93ab-4e90-81cb-3127f7646279" (UID: "d828888d-93ab-4e90-81cb-3127f7646279"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:55:54.934892 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:54.934825    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:55:54.934892 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:54.934844    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:55:54.934892 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:54.934822    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d828888d-93ab-4e90-81cb-3127f7646279" (UID: "d828888d-93ab-4e90-81cb-3127f7646279"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:55:54.936641 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:54.936615    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d828888d-93ab-4e90-81cb-3127f7646279-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d828888d-93ab-4e90-81cb-3127f7646279" (UID: "d828888d-93ab-4e90-81cb-3127f7646279"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:55:54.936641 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:54.936630    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d828888d-93ab-4e90-81cb-3127f7646279-kube-api-access-hfk4n" (OuterVolumeSpecName: "kube-api-access-hfk4n") pod "d828888d-93ab-4e90-81cb-3127f7646279" (UID: "d828888d-93ab-4e90-81cb-3127f7646279"). InnerVolumeSpecName "kube-api-access-hfk4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:55:54.936787 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:54.936634    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-dshm" (OuterVolumeSpecName: "dshm") pod "d828888d-93ab-4e90-81cb-3127f7646279" (UID: "d828888d-93ab-4e90-81cb-3127f7646279"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:55:55.035754 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:55.035719    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:55:55.035754 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:55.035748    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d828888d-93ab-4e90-81cb-3127f7646279-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:55:55.035754 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:55.035758    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfk4n\" (UniqueName: \"kubernetes.io/projected/d828888d-93ab-4e90-81cb-3127f7646279-kube-api-access-hfk4n\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:55:55.035967 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:55.035766    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d828888d-93ab-4e90-81cb-3127f7646279-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:55:55.490610 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:55.490581    2562 generic.go:356] "Generic (PLEG): container finished" podID="d828888d-93ab-4e90-81cb-3127f7646279" containerID="703d153a318d43bc33e8e10e31278decd59d3737a40c6a9b3a7191074db2ab64" exitCode=0
Sep 15 11:55:55.490783 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:55.490616    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g" event={"ID":"d828888d-93ab-4e90-81cb-3127f7646279","Type":"ContainerDied","Data":"703d153a318d43bc33e8e10e31278decd59d3737a40c6a9b3a7191074db2ab64"}
Sep 15 11:55:55.490783 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:55.490626    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"
Sep 15 11:55:55.490783 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:55.490641    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g" event={"ID":"d828888d-93ab-4e90-81cb-3127f7646279","Type":"ContainerDied","Data":"e447373a1df7de3f28335ec8431fcbc619b79621af7f635469454dc2074c9e37"}
Sep 15 11:55:55.490783 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:55.490658    2562 scope.go:117] "RemoveContainer" containerID="703d153a318d43bc33e8e10e31278decd59d3737a40c6a9b3a7191074db2ab64"
Sep 15 11:55:55.499318 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:55.499302    2562 scope.go:117] "RemoveContainer" containerID="703d153a318d43bc33e8e10e31278decd59d3737a40c6a9b3a7191074db2ab64"
Sep 15 11:55:55.499578 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:55:55.499562    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"703d153a318d43bc33e8e10e31278decd59d3737a40c6a9b3a7191074db2ab64\": container with ID starting with 703d153a318d43bc33e8e10e31278decd59d3737a40c6a9b3a7191074db2ab64 not found: ID does not exist" containerID="703d153a318d43bc33e8e10e31278decd59d3737a40c6a9b3a7191074db2ab64"
Sep 15 11:55:55.499627 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:55.499585    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703d153a318d43bc33e8e10e31278decd59d3737a40c6a9b3a7191074db2ab64"} err="failed to get container status \"703d153a318d43bc33e8e10e31278decd59d3737a40c6a9b3a7191074db2ab64\": rpc error: code = NotFound desc = could not find container \"703d153a318d43bc33e8e10e31278decd59d3737a40c6a9b3a7191074db2ab64\": container with ID starting with 703d153a318d43bc33e8e10e31278decd59d3737a40c6a9b3a7191074db2ab64 not found: ID does not exist"
Sep 15 11:55:55.513158 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:55.513127    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"]
Sep 15 11:55:55.516004 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:55.515984    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7efa8b/rate-test-model-7efa8b-kserve-686b49df99-kp92g"]
Sep 15 11:55:55.873474 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:55.873438    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d828888d-93ab-4e90-81cb-3127f7646279" path="/var/lib/kubelet/pods/d828888d-93ab-4e90-81cb-3127f7646279/volumes"
Sep 15 11:55:58.954539 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:58.954504    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:55:59.464238 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:59.464202    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq"
Sep 15 11:55:59.464453 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:55:59.464426    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2"
Sep 15 11:56:16.268983 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.268941    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"]
Sep 15 11:56:16.269581 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.269315    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d828888d-93ab-4e90-81cb-3127f7646279" containerName="main"
Sep 15 11:56:16.269581 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.269329    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d828888d-93ab-4e90-81cb-3127f7646279" containerName="main"
Sep 15 11:56:16.269581 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.269387    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d828888d-93ab-4e90-81cb-3127f7646279" containerName="main"
Sep 15 11:56:16.272301 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.272273    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"
Sep 15 11:56:16.275097 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.275075    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 11:56:16.275677 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.275644    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 11:56:16.275787 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.275721    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bjt69\""
Sep 15 11:56:16.275787 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.275690    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:56:16.275787 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.275722    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d40293-serving-cert\""
Sep 15 11:56:16.285684 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.285622    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"]
Sep 15 11:56:16.420341 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.420307    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c760383-ae98-4d12-814f-04e557d4993a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284\" (UID: \"6c760383-ae98-4d12-814f-04e557d4993a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"
Sep 15 11:56:16.420341 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.420343    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c760383-ae98-4d12-814f-04e557d4993a-metrics-certs\") pod \"maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284\" (UID: \"6c760383-ae98-4d12-814f-04e557d4993a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"
Sep 15 11:56:16.420550 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.420365    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c760383-ae98-4d12-814f-04e557d4993a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284\" (UID: \"6c760383-ae98-4d12-814f-04e557d4993a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"
Sep 15 11:56:16.420550 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.420483    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvt8t\" (UniqueName: \"kubernetes.io/projected/6c760383-ae98-4d12-814f-04e557d4993a-kube-api-access-qvt8t\") pod \"maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284\" (UID: \"6c760383-ae98-4d12-814f-04e557d4993a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"
Sep 15 11:56:16.521712 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.521631    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvt8t\" (UniqueName: \"kubernetes.io/projected/6c760383-ae98-4d12-814f-04e557d4993a-kube-api-access-qvt8t\") pod \"maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284\" (UID: \"6c760383-ae98-4d12-814f-04e557d4993a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"
Sep 15 11:56:16.521853 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.521747    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c760383-ae98-4d12-814f-04e557d4993a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284\" (UID: \"6c760383-ae98-4d12-814f-04e557d4993a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"
Sep 15 11:56:16.521853 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.521777    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c760383-ae98-4d12-814f-04e557d4993a-metrics-certs\") pod \"maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284\" (UID: \"6c760383-ae98-4d12-814f-04e557d4993a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"
Sep 15 11:56:16.521853 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.521805    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c760383-ae98-4d12-814f-04e557d4993a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284\" (UID: \"6c760383-ae98-4d12-814f-04e557d4993a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"
Sep 15 11:56:16.521984 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:56:16.521878    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-serving-cert: secret "maas-api-e2e-ipp-b-d40293-serving-cert" not found
Sep 15 11:56:16.521984 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:56:16.521962    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c760383-ae98-4d12-814f-04e557d4993a-maas-api-tls podName:6c760383-ae98-4d12-814f-04e557d4993a nodeName:}" failed. No retries permitted until 2026-09-15 11:56:17.021941178 +0000 UTC m=+1875.684436348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c760383-ae98-4d12-814f-04e557d4993a-maas-api-tls") pod "maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284" (UID: "6c760383-ae98-4d12-814f-04e557d4993a") : secret "maas-api-e2e-ipp-b-d40293-serving-cert" not found
Sep 15 11:56:16.522609 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.522590    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c760383-ae98-4d12-814f-04e557d4993a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284\" (UID: \"6c760383-ae98-4d12-814f-04e557d4993a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"
Sep 15 11:56:16.524278 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.524260    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c760383-ae98-4d12-814f-04e557d4993a-metrics-certs\") pod \"maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284\" (UID: \"6c760383-ae98-4d12-814f-04e557d4993a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"
Sep 15 11:56:16.529507 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.529488    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvt8t\" (UniqueName: \"kubernetes.io/projected/6c760383-ae98-4d12-814f-04e557d4993a-kube-api-access-qvt8t\") pod \"maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284\" (UID: \"6c760383-ae98-4d12-814f-04e557d4993a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"
Sep 15 11:56:16.869244 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.869212    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv"]
Sep 15 11:56:16.872799 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.872780    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv"
Sep 15 11:56:16.876244 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.876222    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d40293\""
Sep 15 11:56:16.888223 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:16.888202    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv"]
Sep 15 11:56:17.026433 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:17.026400    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b0fe84c-9592-4b04-a7df-572f46552908-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv\" (UID: \"1b0fe84c-9592-4b04-a7df-572f46552908\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv"
Sep 15 11:56:17.026590 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:17.026455    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c760383-ae98-4d12-814f-04e557d4993a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284\" (UID: \"6c760383-ae98-4d12-814f-04e557d4993a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"
Sep 15 11:56:17.026590 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:17.026502    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wrs2\" (UniqueName: \"kubernetes.io/projected/1b0fe84c-9592-4b04-a7df-572f46552908-kube-api-access-4wrs2\") pod \"payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv\" (UID: \"1b0fe84c-9592-4b04-a7df-572f46552908\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv"
Sep 15 11:56:17.028771 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:17.028741    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c760383-ae98-4d12-814f-04e557d4993a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284\" (UID: \"6c760383-ae98-4d12-814f-04e557d4993a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"
Sep 15 11:56:17.127963 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:17.127883    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b0fe84c-9592-4b04-a7df-572f46552908-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv\" (UID: \"1b0fe84c-9592-4b04-a7df-572f46552908\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv"
Sep 15 11:56:17.127963 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:17.127934    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wrs2\" (UniqueName: \"kubernetes.io/projected/1b0fe84c-9592-4b04-a7df-572f46552908-kube-api-access-4wrs2\") pod \"payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv\" (UID: \"1b0fe84c-9592-4b04-a7df-572f46552908\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv"
Sep 15 11:56:17.128595 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:17.128576    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b0fe84c-9592-4b04-a7df-572f46552908-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv\" (UID: \"1b0fe84c-9592-4b04-a7df-572f46552908\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv"
Sep 15 11:56:17.136336 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:17.136309    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wrs2\" (UniqueName: \"kubernetes.io/projected/1b0fe84c-9592-4b04-a7df-572f46552908-kube-api-access-4wrs2\") pod \"payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv\" (UID: \"1b0fe84c-9592-4b04-a7df-572f46552908\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv"
Sep 15 11:56:17.183384 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:17.183358    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv"
Sep 15 11:56:17.185521 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:17.185501    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"
Sep 15 11:56:17.461871 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:17.461848    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv"]
Sep 15 11:56:17.464468 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:56:17.464437    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0fe84c_9592_4b04_a7df_572f46552908.slice/crio-7243c2dcc460aafa8903ba07fc978f34115a04feb6a46ec63ccacce39df61a1e WatchSource:0}: Error finding container 7243c2dcc460aafa8903ba07fc978f34115a04feb6a46ec63ccacce39df61a1e: Status 404 returned error can't find the container with id 7243c2dcc460aafa8903ba07fc978f34115a04feb6a46ec63ccacce39df61a1e
Sep 15 11:56:17.465533 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:17.465514    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"]
Sep 15 11:56:17.467441 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:56:17.467334    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c760383_ae98_4d12_814f_04e557d4993a.slice/crio-d09487aebcbbc60d635868afe39c26def4bebbb07831a39adebd7325363db5b2 WatchSource:0}: Error finding container d09487aebcbbc60d635868afe39c26def4bebbb07831a39adebd7325363db5b2: Status 404 returned error can't find the container with id d09487aebcbbc60d635868afe39c26def4bebbb07831a39adebd7325363db5b2
Sep 15 11:56:17.468789 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:17.468774    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 11:56:17.601927 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:17.601894    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284" event={"ID":"6c760383-ae98-4d12-814f-04e557d4993a","Type":"ContainerStarted","Data":"d09487aebcbbc60d635868afe39c26def4bebbb07831a39adebd7325363db5b2"}
Sep 15 11:56:17.603293 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:17.603267    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv" event={"ID":"1b0fe84c-9592-4b04-a7df-572f46552908","Type":"ContainerStarted","Data":"70d4d9c6b1a0f1094409102473cde465798a91a2857c83bfaa66c42b2ef3812b"}
Sep 15 11:56:17.603416 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:17.603315    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv" event={"ID":"1b0fe84c-9592-4b04-a7df-572f46552908","Type":"ContainerStarted","Data":"7243c2dcc460aafa8903ba07fc978f34115a04feb6a46ec63ccacce39df61a1e"}
Sep 15 11:56:17.603416 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:17.603407    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv"
Sep 15 11:56:17.620714 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:17.620671    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv" podStartSLOduration=1.6206569910000002 podStartE2EDuration="1.620656991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:56:17.619549304 +0000 UTC m=+1876.282044514" watchObservedRunningTime="2026-09-15 11:56:17.620656991 +0000 UTC m=+1876.283152172"
Sep 15 11:56:18.610188 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:18.610152    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284" event={"ID":"6c760383-ae98-4d12-814f-04e557d4993a","Type":"ContainerStarted","Data":"202dd076e0a870732cb71dabcef17401ca80d212fa511ba778732fac3f85aec9"}
Sep 15 11:56:18.610584 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:18.610216    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"
Sep 15 11:56:18.630379 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:18.630336    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284" podStartSLOduration=2.210171796 podStartE2EDuration="2.630323115s" totalImagesPullingTime="420.151319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:56:18.628406719 +0000 UTC m=+1877.290901901" watchObservedRunningTime="2026-09-15 11:56:18.630323115 +0000 UTC m=+1877.292818296"
Sep 15 11:56:21.060404 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:21.060367    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:56:24.620277 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:24.620189    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"
Sep 15 11:56:28.612473 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:56:28.612442    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv"
Sep 15 11:57:07.149570 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:07.149527    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:57:29.154183 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:29.154146    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:57:35.331671 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:35.331637    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq_c1def9c7-31a0-4398-883d-db966eb6741e/payload-processing/0.log"
Sep 15 11:57:42.861988 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:42.861953    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk"]
Sep 15 11:57:42.865506 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:42.865489    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk"
Sep 15 11:57:42.867652 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:42.867631    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-1e87f5\""
Sep 15 11:57:42.867652 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:42.867648    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-1e87f5-dockercfg-8fzlp\""
Sep 15 11:57:42.877592 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:42.877570    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk"]
Sep 15 11:57:42.891064 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:42.891024    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j"]
Sep 15 11:57:42.894204 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:42.894188    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j"
Sep 15 11:57:42.910246 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:42.910220    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j"]
Sep 15 11:57:42.921732 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:42.921707    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13be1a59-42bd-4f9e-b209-a9422b42cc74-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk\" (UID: \"13be1a59-42bd-4f9e-b209-a9422b42cc74\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk"
Sep 15 11:57:42.921855 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:42.921749    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czxhx\" (UniqueName: \"kubernetes.io/projected/de889a12-c258-4b88-9048-460fbfe043e0-kube-api-access-czxhx\") pod \"payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j\" (UID: \"de889a12-c258-4b88-9048-460fbfe043e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j"
Sep 15 11:57:42.921855 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:42.921822    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccnhz\" (UniqueName: \"kubernetes.io/projected/13be1a59-42bd-4f9e-b209-a9422b42cc74-kube-api-access-ccnhz\") pod \"payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk\" (UID: \"13be1a59-42bd-4f9e-b209-a9422b42cc74\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk"
Sep 15 11:57:42.921855 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:42.921844    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de889a12-c258-4b88-9048-460fbfe043e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j\" (UID: \"de889a12-c258-4b88-9048-460fbfe043e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j"
Sep 15 11:57:43.022299 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:43.022264    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13be1a59-42bd-4f9e-b209-a9422b42cc74-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk\" (UID: \"13be1a59-42bd-4f9e-b209-a9422b42cc74\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk"
Sep 15 11:57:43.022299 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:43.022304    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czxhx\" (UniqueName: \"kubernetes.io/projected/de889a12-c258-4b88-9048-460fbfe043e0-kube-api-access-czxhx\") pod \"payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j\" (UID: \"de889a12-c258-4b88-9048-460fbfe043e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j"
Sep 15 11:57:43.022538 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:43.022344    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccnhz\" (UniqueName: \"kubernetes.io/projected/13be1a59-42bd-4f9e-b209-a9422b42cc74-kube-api-access-ccnhz\") pod \"payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk\" (UID: \"13be1a59-42bd-4f9e-b209-a9422b42cc74\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk"
Sep 15 11:57:43.022538 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:43.022365    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de889a12-c258-4b88-9048-460fbfe043e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j\" (UID: \"de889a12-c258-4b88-9048-460fbfe043e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j"
Sep 15 11:57:43.023029 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:43.023003    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13be1a59-42bd-4f9e-b209-a9422b42cc74-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk\" (UID: \"13be1a59-42bd-4f9e-b209-a9422b42cc74\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk"
Sep 15 11:57:43.023144 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:43.023027    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de889a12-c258-4b88-9048-460fbfe043e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j\" (UID: \"de889a12-c258-4b88-9048-460fbfe043e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j"
Sep 15 11:57:43.030235 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:43.030205    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccnhz\" (UniqueName: \"kubernetes.io/projected/13be1a59-42bd-4f9e-b209-a9422b42cc74-kube-api-access-ccnhz\") pod \"payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk\" (UID: \"13be1a59-42bd-4f9e-b209-a9422b42cc74\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk"
Sep 15 11:57:43.030344 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:43.030258    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czxhx\" (UniqueName: \"kubernetes.io/projected/de889a12-c258-4b88-9048-460fbfe043e0-kube-api-access-czxhx\") pod \"payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j\" (UID: \"de889a12-c258-4b88-9048-460fbfe043e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j"
Sep 15 11:57:43.176286 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:43.176199    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk"
Sep 15 11:57:43.208569 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:43.208541    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j"
Sep 15 11:57:43.452547 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:43.452471    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk"]
Sep 15 11:57:43.455772 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:57:43.455745    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13be1a59_42bd_4f9e_b209_a9422b42cc74.slice/crio-edcbd40908b6b7ecef8dfa7b2667db6830583211a04771ff8cc8d12c3acad0f6 WatchSource:0}: Error finding container edcbd40908b6b7ecef8dfa7b2667db6830583211a04771ff8cc8d12c3acad0f6: Status 404 returned error can't find the container with id edcbd40908b6b7ecef8dfa7b2667db6830583211a04771ff8cc8d12c3acad0f6
Sep 15 11:57:43.493581 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:43.493546    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j"]
Sep 15 11:57:43.495722 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:57:43.495699    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde889a12_c258_4b88_9048_460fbfe043e0.slice/crio-eec98c370427b31ab3882a0584de8e9741e12a4c46e776a0fac88139d330bb5b WatchSource:0}: Error finding container eec98c370427b31ab3882a0584de8e9741e12a4c46e776a0fac88139d330bb5b: Status 404 returned error can't find the container with id eec98c370427b31ab3882a0584de8e9741e12a4c46e776a0fac88139d330bb5b
Sep 15 11:57:44.002970 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:44.002934    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk" event={"ID":"13be1a59-42bd-4f9e-b209-a9422b42cc74","Type":"ContainerStarted","Data":"d3ad081742771b1682682e5aed03e7661711208213b1ad35171584548728c0ee"}
Sep 15 11:57:44.003451 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:44.002997    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk"
Sep 15 11:57:44.003451 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:44.003023    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk" event={"ID":"13be1a59-42bd-4f9e-b209-a9422b42cc74","Type":"ContainerStarted","Data":"edcbd40908b6b7ecef8dfa7b2667db6830583211a04771ff8cc8d12c3acad0f6"}
Sep 15 11:57:44.004696 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:44.004669    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j" event={"ID":"de889a12-c258-4b88-9048-460fbfe043e0","Type":"ContainerStarted","Data":"6de90f3b30797938c5175955761ba492c5b24c2c807e1617afa87dfb7b278ebc"}
Sep 15 11:57:44.004814 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:44.004712    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j" event={"ID":"de889a12-c258-4b88-9048-460fbfe043e0","Type":"ContainerStarted","Data":"eec98c370427b31ab3882a0584de8e9741e12a4c46e776a0fac88139d330bb5b"}
Sep 15 11:57:44.004814 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:44.004801    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j"
Sep 15 11:57:44.021634 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:44.021589    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk" podStartSLOduration=2.021574503 podStartE2EDuration="2.021574503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:57:44.02096645 +0000 UTC m=+1962.683461644" watchObservedRunningTime="2026-09-15 11:57:44.021574503 +0000 UTC m=+1962.684069683"
Sep 15 11:57:44.040209 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:44.040161    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j" podStartSLOduration=2.040145744 podStartE2EDuration="2.040145744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:57:44.038277352 +0000 UTC m=+1962.700772551" watchObservedRunningTime="2026-09-15 11:57:44.040145744 +0000 UTC m=+1962.702640926"
Sep 15 11:57:49.242866 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:49.242831    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:57:55.013297 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:55.013211    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk"
Sep 15 11:57:55.013742 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:57:55.013407    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j"
Sep 15 11:58:02.139643 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:02.139599    2562 scope.go:117] "RemoveContainer" containerID="073cd3896b5882a5f84b16bf1bcdd3db2050524687f58de9776608cb1bf5c4f3"
Sep 15 11:58:09.814671 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:09.814635    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1e87f5-z9skp"]
Sep 15 11:58:09.818101 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:09.818082    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1e87f5-z9skp"
Sep 15 11:58:09.820234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:09.820212    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rbxjk\""
Sep 15 11:58:09.825848 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:09.825825    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1e87f5-z9skp"]
Sep 15 11:58:09.942193 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:09.942143    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a3fbf30-352d-4947-8fdb-494669f6f6c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1e87f5-z9skp\" (UID: \"8a3fbf30-352d-4947-8fdb-494669f6f6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1e87f5-z9skp"
Sep 15 11:58:10.043404 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:10.043350    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a3fbf30-352d-4947-8fdb-494669f6f6c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1e87f5-z9skp\" (UID: \"8a3fbf30-352d-4947-8fdb-494669f6f6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1e87f5-z9skp"
Sep 15 11:58:10.043984 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:10.043963    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a3fbf30-352d-4947-8fdb-494669f6f6c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1e87f5-z9skp\" (UID: \"8a3fbf30-352d-4947-8fdb-494669f6f6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1e87f5-z9skp"
Sep 15 11:58:10.128607 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:10.128580    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1e87f5-z9skp"
Sep 15 11:58:10.403808 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:10.403784    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1e87f5-z9skp"]
Sep 15 11:58:10.406297 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:58:10.406271    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3fbf30_352d_4947_8fdb_494669f6f6c4.slice/crio-58dfa2d3d16579f9b72d769c3a401f7fcdc15c8cac2467ef8b95919039db8696 WatchSource:0}: Error finding container 58dfa2d3d16579f9b72d769c3a401f7fcdc15c8cac2467ef8b95919039db8696: Status 404 returned error can't find the container with id 58dfa2d3d16579f9b72d769c3a401f7fcdc15c8cac2467ef8b95919039db8696
Sep 15 11:58:11.141362 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:11.141328    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1e87f5-z9skp" event={"ID":"8a3fbf30-352d-4947-8fdb-494669f6f6c4","Type":"ContainerStarted","Data":"540ca4fdddcfb44d618c61c5a75fda847eb070c6edaf09d83546450637fedd1d"}
Sep 15 11:58:11.141723 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:11.141388    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1e87f5-z9skp" event={"ID":"8a3fbf30-352d-4947-8fdb-494669f6f6c4","Type":"ContainerStarted","Data":"58dfa2d3d16579f9b72d769c3a401f7fcdc15c8cac2467ef8b95919039db8696"}
Sep 15 11:58:11.159986 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:11.159926    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1e87f5-z9skp" podStartSLOduration=2.159907511 podStartE2EDuration="2.159907511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:58:11.159265326 +0000 UTC m=+1989.821760508" watchObservedRunningTime="2026-09-15 11:58:11.159907511 +0000 UTC m=+1989.822402693"
Sep 15 11:58:12.147122 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:12.147088    2562 generic.go:356] "Generic (PLEG): container finished" podID="8a3fbf30-352d-4947-8fdb-494669f6f6c4" containerID="540ca4fdddcfb44d618c61c5a75fda847eb070c6edaf09d83546450637fedd1d" exitCode=0
Sep 15 11:58:12.147622 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:12.147218    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1e87f5-z9skp" event={"ID":"8a3fbf30-352d-4947-8fdb-494669f6f6c4","Type":"ContainerDied","Data":"540ca4fdddcfb44d618c61c5a75fda847eb070c6edaf09d83546450637fedd1d"}
Sep 15 11:58:13.288298 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:13.288271    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1e87f5-z9skp"
Sep 15 11:58:13.372518 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:13.372482    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a3fbf30-352d-4947-8fdb-494669f6f6c4-maas-api-service-ca\") pod \"8a3fbf30-352d-4947-8fdb-494669f6f6c4\" (UID: \"8a3fbf30-352d-4947-8fdb-494669f6f6c4\") "
Sep 15 11:58:13.372847 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:13.372823    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3fbf30-352d-4947-8fdb-494669f6f6c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8a3fbf30-352d-4947-8fdb-494669f6f6c4" (UID: "8a3fbf30-352d-4947-8fdb-494669f6f6c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:58:13.473687 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:13.473588    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a3fbf30-352d-4947-8fdb-494669f6f6c4-maas-api-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:58:14.158104 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:14.158015    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1e87f5-z9skp"
Sep 15 11:58:14.158248 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:14.158018    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1e87f5-z9skp" event={"ID":"8a3fbf30-352d-4947-8fdb-494669f6f6c4","Type":"ContainerDied","Data":"58dfa2d3d16579f9b72d769c3a401f7fcdc15c8cac2467ef8b95919039db8696"}
Sep 15 11:58:14.158248 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:14.158145    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58dfa2d3d16579f9b72d769c3a401f7fcdc15c8cac2467ef8b95919039db8696"
Sep 15 11:58:20.853920 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:20.853881    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j"]
Sep 15 11:58:20.854447 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:20.854160    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j" podUID="de889a12-c258-4b88-9048-460fbfe043e0" containerName="payload-processing" containerID="cri-o://6de90f3b30797938c5175955761ba492c5b24c2c807e1617afa87dfb7b278ebc" gracePeriod=30
Sep 15 11:58:20.863144 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:20.863118    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk"]
Sep 15 11:58:20.863372 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:20.863344    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk" podUID="13be1a59-42bd-4f9e-b209-a9422b42cc74" containerName="payload-pre-processing" containerID="cri-o://d3ad081742771b1682682e5aed03e7661711208213b1ad35171584548728c0ee" gracePeriod=30
Sep 15 11:58:21.036628 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.036566    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1e87f5-z9skp"]
Sep 15 11:58:21.041428 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.041400    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1e87f5-z9skp"]
Sep 15 11:58:21.141196 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.141165    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j"
Sep 15 11:58:21.144226 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.144203    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk"
Sep 15 11:58:21.191132 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.191095    2562 generic.go:356] "Generic (PLEG): container finished" podID="13be1a59-42bd-4f9e-b209-a9422b42cc74" containerID="d3ad081742771b1682682e5aed03e7661711208213b1ad35171584548728c0ee" exitCode=0
Sep 15 11:58:21.191132 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.191126    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk"
Sep 15 11:58:21.191401 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.191168    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk" event={"ID":"13be1a59-42bd-4f9e-b209-a9422b42cc74","Type":"ContainerDied","Data":"d3ad081742771b1682682e5aed03e7661711208213b1ad35171584548728c0ee"}
Sep 15 11:58:21.191401 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.191230    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk" event={"ID":"13be1a59-42bd-4f9e-b209-a9422b42cc74","Type":"ContainerDied","Data":"edcbd40908b6b7ecef8dfa7b2667db6830583211a04771ff8cc8d12c3acad0f6"}
Sep 15 11:58:21.191401 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.191263    2562 scope.go:117] "RemoveContainer" containerID="d3ad081742771b1682682e5aed03e7661711208213b1ad35171584548728c0ee"
Sep 15 11:58:21.192520 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.192499    2562 generic.go:356] "Generic (PLEG): container finished" podID="de889a12-c258-4b88-9048-460fbfe043e0" containerID="6de90f3b30797938c5175955761ba492c5b24c2c807e1617afa87dfb7b278ebc" exitCode=0
Sep 15 11:58:21.192627 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.192537    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j"
Sep 15 11:58:21.192627 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.192572    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j" event={"ID":"de889a12-c258-4b88-9048-460fbfe043e0","Type":"ContainerDied","Data":"6de90f3b30797938c5175955761ba492c5b24c2c807e1617afa87dfb7b278ebc"}
Sep 15 11:58:21.192708 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.192624    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j" event={"ID":"de889a12-c258-4b88-9048-460fbfe043e0","Type":"ContainerDied","Data":"eec98c370427b31ab3882a0584de8e9741e12a4c46e776a0fac88139d330bb5b"}
Sep 15 11:58:21.201012 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.200959    2562 scope.go:117] "RemoveContainer" containerID="d3ad081742771b1682682e5aed03e7661711208213b1ad35171584548728c0ee"
Sep 15 11:58:21.201390 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:58:21.201360    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3ad081742771b1682682e5aed03e7661711208213b1ad35171584548728c0ee\": container with ID starting with d3ad081742771b1682682e5aed03e7661711208213b1ad35171584548728c0ee not found: ID does not exist" containerID="d3ad081742771b1682682e5aed03e7661711208213b1ad35171584548728c0ee"
Sep 15 11:58:21.201509 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.201403    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3ad081742771b1682682e5aed03e7661711208213b1ad35171584548728c0ee"} err="failed to get container status \"d3ad081742771b1682682e5aed03e7661711208213b1ad35171584548728c0ee\": rpc error: code = NotFound desc = could not find container \"d3ad081742771b1682682e5aed03e7661711208213b1ad35171584548728c0ee\": container with ID starting with d3ad081742771b1682682e5aed03e7661711208213b1ad35171584548728c0ee not found: ID does not exist"
Sep 15 11:58:21.201509 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.201430    2562 scope.go:117] "RemoveContainer" containerID="6de90f3b30797938c5175955761ba492c5b24c2c807e1617afa87dfb7b278ebc"
Sep 15 11:58:21.211313 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.211285    2562 scope.go:117] "RemoveContainer" containerID="6de90f3b30797938c5175955761ba492c5b24c2c807e1617afa87dfb7b278ebc"
Sep 15 11:58:21.211709 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:58:21.211682    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6de90f3b30797938c5175955761ba492c5b24c2c807e1617afa87dfb7b278ebc\": container with ID starting with 6de90f3b30797938c5175955761ba492c5b24c2c807e1617afa87dfb7b278ebc not found: ID does not exist" containerID="6de90f3b30797938c5175955761ba492c5b24c2c807e1617afa87dfb7b278ebc"
Sep 15 11:58:21.211827 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.211715    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de90f3b30797938c5175955761ba492c5b24c2c807e1617afa87dfb7b278ebc"} err="failed to get container status \"6de90f3b30797938c5175955761ba492c5b24c2c807e1617afa87dfb7b278ebc\": rpc error: code = NotFound desc = could not find container \"6de90f3b30797938c5175955761ba492c5b24c2c807e1617afa87dfb7b278ebc\": container with ID starting with 6de90f3b30797938c5175955761ba492c5b24c2c807e1617afa87dfb7b278ebc not found: ID does not exist"
Sep 15 11:58:21.233109 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.233076    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccnhz\" (UniqueName: \"kubernetes.io/projected/13be1a59-42bd-4f9e-b209-a9422b42cc74-kube-api-access-ccnhz\") pod \"13be1a59-42bd-4f9e-b209-a9422b42cc74\" (UID: \"13be1a59-42bd-4f9e-b209-a9422b42cc74\") "
Sep 15 11:58:21.233260 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.233119    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13be1a59-42bd-4f9e-b209-a9422b42cc74-plugins-config-volume\") pod \"13be1a59-42bd-4f9e-b209-a9422b42cc74\" (UID: \"13be1a59-42bd-4f9e-b209-a9422b42cc74\") "
Sep 15 11:58:21.233260 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.233169    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de889a12-c258-4b88-9048-460fbfe043e0-plugins-config-volume\") pod \"de889a12-c258-4b88-9048-460fbfe043e0\" (UID: \"de889a12-c258-4b88-9048-460fbfe043e0\") "
Sep 15 11:58:21.233260 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.233245    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czxhx\" (UniqueName: \"kubernetes.io/projected/de889a12-c258-4b88-9048-460fbfe043e0-kube-api-access-czxhx\") pod \"de889a12-c258-4b88-9048-460fbfe043e0\" (UID: \"de889a12-c258-4b88-9048-460fbfe043e0\") "
Sep 15 11:58:21.233557 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.233532    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13be1a59-42bd-4f9e-b209-a9422b42cc74-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13be1a59-42bd-4f9e-b209-a9422b42cc74" (UID: "13be1a59-42bd-4f9e-b209-a9422b42cc74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:58:21.233633 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.233591    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de889a12-c258-4b88-9048-460fbfe043e0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "de889a12-c258-4b88-9048-460fbfe043e0" (UID: "de889a12-c258-4b88-9048-460fbfe043e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:58:21.235402 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.235365    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13be1a59-42bd-4f9e-b209-a9422b42cc74-kube-api-access-ccnhz" (OuterVolumeSpecName: "kube-api-access-ccnhz") pod "13be1a59-42bd-4f9e-b209-a9422b42cc74" (UID: "13be1a59-42bd-4f9e-b209-a9422b42cc74"). InnerVolumeSpecName "kube-api-access-ccnhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:58:21.235402 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.235385    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de889a12-c258-4b88-9048-460fbfe043e0-kube-api-access-czxhx" (OuterVolumeSpecName: "kube-api-access-czxhx") pod "de889a12-c258-4b88-9048-460fbfe043e0" (UID: "de889a12-c258-4b88-9048-460fbfe043e0"). InnerVolumeSpecName "kube-api-access-czxhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:58:21.334801 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.334749    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de889a12-c258-4b88-9048-460fbfe043e0-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:58:21.334801 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.334788    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czxhx\" (UniqueName: \"kubernetes.io/projected/de889a12-c258-4b88-9048-460fbfe043e0-kube-api-access-czxhx\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:58:21.334801 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.334798    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccnhz\" (UniqueName: \"kubernetes.io/projected/13be1a59-42bd-4f9e-b209-a9422b42cc74-kube-api-access-ccnhz\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:58:21.334801 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.334806    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13be1a59-42bd-4f9e-b209-a9422b42cc74-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:58:21.516007 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.515968    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j"]
Sep 15 11:58:21.522433 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.522403    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1e87f5-7c565789b7-xtp2j"]
Sep 15 11:58:21.535294 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.535259    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk"]
Sep 15 11:58:21.538216 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.538183    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1e87f5-d76885474-2g8qk"]
Sep 15 11:58:21.873223 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.873194    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13be1a59-42bd-4f9e-b209-a9422b42cc74" path="/var/lib/kubelet/pods/13be1a59-42bd-4f9e-b209-a9422b42cc74/volumes"
Sep 15 11:58:21.873590 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.873511    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a3fbf30-352d-4947-8fdb-494669f6f6c4" path="/var/lib/kubelet/pods/8a3fbf30-352d-4947-8fdb-494669f6f6c4/volumes"
Sep 15 11:58:21.873816 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:21.873803    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de889a12-c258-4b88-9048-460fbfe043e0" path="/var/lib/kubelet/pods/de889a12-c258-4b88-9048-460fbfe043e0/volumes"
Sep 15 11:58:25.458245 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:25.458210    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:58:30.819863 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:30.819828    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-173324-jrrcc"]
Sep 15 11:58:30.820445 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:30.820369    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de889a12-c258-4b88-9048-460fbfe043e0" containerName="payload-processing"
Sep 15 11:58:30.820445 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:30.820390    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="de889a12-c258-4b88-9048-460fbfe043e0" containerName="payload-processing"
Sep 15 11:58:30.820445 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:30.820410    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3fbf30-352d-4947-8fdb-494669f6f6c4" containerName="revoke-keys"
Sep 15 11:58:30.820445 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:30.820419    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3fbf30-352d-4947-8fdb-494669f6f6c4" containerName="revoke-keys"
Sep 15 11:58:30.820445 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:30.820433    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13be1a59-42bd-4f9e-b209-a9422b42cc74" containerName="payload-pre-processing"
Sep 15 11:58:30.820445 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:30.820440    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="13be1a59-42bd-4f9e-b209-a9422b42cc74" containerName="payload-pre-processing"
Sep 15 11:58:30.820765 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:30.820547    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="13be1a59-42bd-4f9e-b209-a9422b42cc74" containerName="payload-pre-processing"
Sep 15 11:58:30.820765 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:30.820563    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3fbf30-352d-4947-8fdb-494669f6f6c4" containerName="revoke-keys"
Sep 15 11:58:30.820765 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:30.820572    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="de889a12-c258-4b88-9048-460fbfe043e0" containerName="payload-processing"
Sep 15 11:58:30.825307 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:30.825286    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-173324-jrrcc"
Sep 15 11:58:30.827500 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:30.827480    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rbxjk\""
Sep 15 11:58:30.830694 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:30.830658    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-173324-jrrcc"]
Sep 15 11:58:30.918443 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:30.918413    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8591083d-c1e6-433b-961b-6c6067ee6734-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-173324-jrrcc\" (UID: \"8591083d-c1e6-433b-961b-6c6067ee6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-173324-jrrcc"
Sep 15 11:58:31.019393 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:31.019365    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8591083d-c1e6-433b-961b-6c6067ee6734-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-173324-jrrcc\" (UID: \"8591083d-c1e6-433b-961b-6c6067ee6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-173324-jrrcc"
Sep 15 11:58:31.019918 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:31.019899    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8591083d-c1e6-433b-961b-6c6067ee6734-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-173324-jrrcc\" (UID: \"8591083d-c1e6-433b-961b-6c6067ee6734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-173324-jrrcc"
Sep 15 11:58:31.138322 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:31.138294    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-173324-jrrcc"
Sep 15 11:58:31.417400 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:31.417350    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-173324-jrrcc"]
Sep 15 11:58:31.420394 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:58:31.420366    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8591083d_c1e6_433b_961b_6c6067ee6734.slice/crio-d702eba00a53ba553b549fb2a922be731ced63570a817b352bacaae002871cca WatchSource:0}: Error finding container d702eba00a53ba553b549fb2a922be731ced63570a817b352bacaae002871cca: Status 404 returned error can't find the container with id d702eba00a53ba553b549fb2a922be731ced63570a817b352bacaae002871cca
Sep 15 11:58:32.248476 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:32.248438    2562 generic.go:356] "Generic (PLEG): container finished" podID="8591083d-c1e6-433b-961b-6c6067ee6734" containerID="fd6df5b0764d003529728415b37162d2967dd8ac29c6481bcc0124fbd058e88c" exitCode=0
Sep 15 11:58:32.248831 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:32.248496    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-173324-jrrcc" event={"ID":"8591083d-c1e6-433b-961b-6c6067ee6734","Type":"ContainerDied","Data":"fd6df5b0764d003529728415b37162d2967dd8ac29c6481bcc0124fbd058e88c"}
Sep 15 11:58:32.248831 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:32.248537    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-173324-jrrcc" event={"ID":"8591083d-c1e6-433b-961b-6c6067ee6734","Type":"ContainerStarted","Data":"d702eba00a53ba553b549fb2a922be731ced63570a817b352bacaae002871cca"}
Sep 15 11:58:33.380628 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:33.380608    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-173324-jrrcc"
Sep 15 11:58:33.538635 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:33.538550    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8591083d-c1e6-433b-961b-6c6067ee6734-maas-api-service-ca\") pod \"8591083d-c1e6-433b-961b-6c6067ee6734\" (UID: \"8591083d-c1e6-433b-961b-6c6067ee6734\") "
Sep 15 11:58:33.538931 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:33.538909    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8591083d-c1e6-433b-961b-6c6067ee6734-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8591083d-c1e6-433b-961b-6c6067ee6734" (UID: "8591083d-c1e6-433b-961b-6c6067ee6734"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:58:33.639449 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:33.639417    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8591083d-c1e6-433b-961b-6c6067ee6734-maas-api-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:58:34.258972 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:34.258893    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-173324-jrrcc"
Sep 15 11:58:34.259135 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:34.258894    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-173324-jrrcc" event={"ID":"8591083d-c1e6-433b-961b-6c6067ee6734","Type":"ContainerDied","Data":"d702eba00a53ba553b549fb2a922be731ced63570a817b352bacaae002871cca"}
Sep 15 11:58:34.259135 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:34.259020    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d702eba00a53ba553b549fb2a922be731ced63570a817b352bacaae002871cca"
Sep 15 11:58:43.849697 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:43.849667    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:58:46.312017 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:46.311984    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq"]
Sep 15 11:58:46.312511 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:46.312206    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq" podUID="c1def9c7-31a0-4398-883d-db966eb6741e" containerName="payload-processing" containerID="cri-o://a1657f6186404f5c70a47a5400c20dd3182da77e473854441b6b0a002e7ab824" gracePeriod=30
Sep 15 11:58:46.319755 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:46.319729    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2"]
Sep 15 11:58:46.321033 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:46.320647    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2" podUID="3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f" containerName="payload-pre-processing" containerID="cri-o://ebe6c452f0ff220571610897a16501711f55a8e570d1889fdc6916590ed5136e" gracePeriod=30
Sep 15 11:58:46.496047 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:46.496013    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-173324-jrrcc"]
Sep 15 11:58:46.502683 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:46.502646    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-173324-jrrcc"]
Sep 15 11:58:46.593803 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:46.593776    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq"
Sep 15 11:58:46.596488 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:46.596471    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2"
Sep 15 11:58:46.651624 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:46.651579    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-648l9\" (UniqueName: \"kubernetes.io/projected/c1def9c7-31a0-4398-883d-db966eb6741e-kube-api-access-648l9\") pod \"c1def9c7-31a0-4398-883d-db966eb6741e\" (UID: \"c1def9c7-31a0-4398-883d-db966eb6741e\") "
Sep 15 11:58:46.651794 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:46.651678    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlqfl\" (UniqueName: \"kubernetes.io/projected/3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f-kube-api-access-jlqfl\") pod \"3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f\" (UID: \"3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f\") "
Sep 15 11:58:46.651794 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:46.651708    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f-plugins-config-volume\") pod \"3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f\" (UID: \"3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f\") "
Sep 15 11:58:46.651794 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:46.651770    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1def9c7-31a0-4398-883d-db966eb6741e-plugins-config-volume\") pod \"c1def9c7-31a0-4398-883d-db966eb6741e\" (UID: \"c1def9c7-31a0-4398-883d-db966eb6741e\") "
Sep 15 11:58:46.652167 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:46.652144    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f" (UID: "3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:58:46.652235 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:46.652186    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1def9c7-31a0-4398-883d-db966eb6741e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1def9c7-31a0-4398-883d-db966eb6741e" (UID: "c1def9c7-31a0-4398-883d-db966eb6741e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:58:46.653621 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:46.653603    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1def9c7-31a0-4398-883d-db966eb6741e-kube-api-access-648l9" (OuterVolumeSpecName: "kube-api-access-648l9") pod "c1def9c7-31a0-4398-883d-db966eb6741e" (UID: "c1def9c7-31a0-4398-883d-db966eb6741e"). InnerVolumeSpecName "kube-api-access-648l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:58:46.653706 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:46.653687    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f-kube-api-access-jlqfl" (OuterVolumeSpecName: "kube-api-access-jlqfl") pod "3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f" (UID: "3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f"). InnerVolumeSpecName "kube-api-access-jlqfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:58:46.752608 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:46.752573    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-648l9\" (UniqueName: \"kubernetes.io/projected/c1def9c7-31a0-4398-883d-db966eb6741e-kube-api-access-648l9\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:58:46.752608 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:46.752600    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlqfl\" (UniqueName: \"kubernetes.io/projected/3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f-kube-api-access-jlqfl\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:58:46.752608 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:46.752609    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:58:46.752828 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:46.752619    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1def9c7-31a0-4398-883d-db966eb6741e-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:58:47.324440 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.324404    2562 generic.go:356] "Generic (PLEG): container finished" podID="3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f" containerID="ebe6c452f0ff220571610897a16501711f55a8e570d1889fdc6916590ed5136e" exitCode=0
Sep 15 11:58:47.324918 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.324450    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2"
Sep 15 11:58:47.324918 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.324489    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2" event={"ID":"3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f","Type":"ContainerDied","Data":"ebe6c452f0ff220571610897a16501711f55a8e570d1889fdc6916590ed5136e"}
Sep 15 11:58:47.324918 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.324541    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2" event={"ID":"3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f","Type":"ContainerDied","Data":"8a3e60e1a2f68c98f068f152075f1a0698f2f605365f197956a879ce3f687026"}
Sep 15 11:58:47.324918 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.324562    2562 scope.go:117] "RemoveContainer" containerID="ebe6c452f0ff220571610897a16501711f55a8e570d1889fdc6916590ed5136e"
Sep 15 11:58:47.326040 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.326014    2562 generic.go:356] "Generic (PLEG): container finished" podID="c1def9c7-31a0-4398-883d-db966eb6741e" containerID="a1657f6186404f5c70a47a5400c20dd3182da77e473854441b6b0a002e7ab824" exitCode=0
Sep 15 11:58:47.326186 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.326073    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq"
Sep 15 11:58:47.326186 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.326113    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq" event={"ID":"c1def9c7-31a0-4398-883d-db966eb6741e","Type":"ContainerDied","Data":"a1657f6186404f5c70a47a5400c20dd3182da77e473854441b6b0a002e7ab824"}
Sep 15 11:58:47.326306 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.326185    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq" event={"ID":"c1def9c7-31a0-4398-883d-db966eb6741e","Type":"ContainerDied","Data":"cb3dd4d40ac4c3ed8515485e0199c505052d622f69283df8e7b9d382698bc19b"}
Sep 15 11:58:47.338196 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.338180    2562 scope.go:117] "RemoveContainer" containerID="ebe6c452f0ff220571610897a16501711f55a8e570d1889fdc6916590ed5136e"
Sep 15 11:58:47.338443 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:58:47.338425    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe6c452f0ff220571610897a16501711f55a8e570d1889fdc6916590ed5136e\": container with ID starting with ebe6c452f0ff220571610897a16501711f55a8e570d1889fdc6916590ed5136e not found: ID does not exist" containerID="ebe6c452f0ff220571610897a16501711f55a8e570d1889fdc6916590ed5136e"
Sep 15 11:58:47.338496 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.338453    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe6c452f0ff220571610897a16501711f55a8e570d1889fdc6916590ed5136e"} err="failed to get container status \"ebe6c452f0ff220571610897a16501711f55a8e570d1889fdc6916590ed5136e\": rpc error: code = NotFound desc = could not find container \"ebe6c452f0ff220571610897a16501711f55a8e570d1889fdc6916590ed5136e\": container with ID starting with ebe6c452f0ff220571610897a16501711f55a8e570d1889fdc6916590ed5136e not found: ID does not exist"
Sep 15 11:58:47.338496 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.338469    2562 scope.go:117] "RemoveContainer" containerID="a1657f6186404f5c70a47a5400c20dd3182da77e473854441b6b0a002e7ab824"
Sep 15 11:58:47.346771 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.346749    2562 scope.go:117] "RemoveContainer" containerID="a1657f6186404f5c70a47a5400c20dd3182da77e473854441b6b0a002e7ab824"
Sep 15 11:58:47.347039 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:58:47.347018    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1657f6186404f5c70a47a5400c20dd3182da77e473854441b6b0a002e7ab824\": container with ID starting with a1657f6186404f5c70a47a5400c20dd3182da77e473854441b6b0a002e7ab824 not found: ID does not exist" containerID="a1657f6186404f5c70a47a5400c20dd3182da77e473854441b6b0a002e7ab824"
Sep 15 11:58:47.347165 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.347045    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1657f6186404f5c70a47a5400c20dd3182da77e473854441b6b0a002e7ab824"} err="failed to get container status \"a1657f6186404f5c70a47a5400c20dd3182da77e473854441b6b0a002e7ab824\": rpc error: code = NotFound desc = could not find container \"a1657f6186404f5c70a47a5400c20dd3182da77e473854441b6b0a002e7ab824\": container with ID starting with a1657f6186404f5c70a47a5400c20dd3182da77e473854441b6b0a002e7ab824 not found: ID does not exist"
Sep 15 11:58:47.349671 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.349650    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2"]
Sep 15 11:58:47.353991 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.353970    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-173324-7bd485c4c4-zxdz2"]
Sep 15 11:58:47.363202 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.363181    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq"]
Sep 15 11:58:47.367041 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.367021    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-173324-7bb57d89bf-gcrbq"]
Sep 15 11:58:47.873578 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.873548    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f" path="/var/lib/kubelet/pods/3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f/volumes"
Sep 15 11:58:47.873879 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.873867    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8591083d-c1e6-433b-961b-6c6067ee6734" path="/var/lib/kubelet/pods/8591083d-c1e6-433b-961b-6c6067ee6734/volumes"
Sep 15 11:58:47.874217 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:47.874203    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1def9c7-31a0-4398-883d-db966eb6741e" path="/var/lib/kubelet/pods/c1def9c7-31a0-4398-883d-db966eb6741e/volumes"
Sep 15 11:58:57.140430 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:58:57.140399    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:59:02.158372 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:02.158342    2562 scope.go:117] "RemoveContainer" containerID="7fc94dd932844bd4a75698f373f3f641f96e3a570a9c896d98f219ba876cf4b1"
Sep 15 11:59:02.166738 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:02.166715    2562 scope.go:117] "RemoveContainer" containerID="7f217ba0b35329e39dbfc22e31578d6df17adf1f11e2653e516ff54f590f8d41"
Sep 15 11:59:35.527799 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:35.527708    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d40293-wbs89"]
Sep 15 11:59:35.528383 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:35.528218    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1def9c7-31a0-4398-883d-db966eb6741e" containerName="payload-processing"
Sep 15 11:59:35.528383 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:35.528240    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1def9c7-31a0-4398-883d-db966eb6741e" containerName="payload-processing"
Sep 15 11:59:35.528383 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:35.528260    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f" containerName="payload-pre-processing"
Sep 15 11:59:35.528383 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:35.528268    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f" containerName="payload-pre-processing"
Sep 15 11:59:35.528383 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:35.528283    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8591083d-c1e6-433b-961b-6c6067ee6734" containerName="revoke-keys"
Sep 15 11:59:35.528383 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:35.528294    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8591083d-c1e6-433b-961b-6c6067ee6734" containerName="revoke-keys"
Sep 15 11:59:35.528700 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:35.528409    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1def9c7-31a0-4398-883d-db966eb6741e" containerName="payload-processing"
Sep 15 11:59:35.528700 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:35.528423    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8591083d-c1e6-433b-961b-6c6067ee6734" containerName="revoke-keys"
Sep 15 11:59:35.528700 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:35.528433    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3058e8c1-f6c4-4c7f-b2d1-dbb5f098a10f" containerName="payload-pre-processing"
Sep 15 11:59:35.531738 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:35.531715    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d40293-wbs89"
Sep 15 11:59:35.534303 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:35.534283    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rbxjk\""
Sep 15 11:59:35.539897 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:35.539845    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d40293-wbs89"]
Sep 15 11:59:35.641271 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:35.641237    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d40293-wbs89\" (UID: \"aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d40293-wbs89"
Sep 15 11:59:35.742079 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:35.742014    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d40293-wbs89\" (UID: \"aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d40293-wbs89"
Sep 15 11:59:35.742621 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:35.742602    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d40293-wbs89\" (UID: \"aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d40293-wbs89"
Sep 15 11:59:35.841948 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:35.841913    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d40293-wbs89"
Sep 15 11:59:36.118466 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:36.118377    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d40293-wbs89"]
Sep 15 11:59:36.120983 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:59:36.120953    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaefe90bc_ff92_43d9_a9c5_bac8fcfeaf15.slice/crio-426ce80af7f6e85325d2c1b95088b98c190dce3f5061e967b65627ea620bf8e7 WatchSource:0}: Error finding container 426ce80af7f6e85325d2c1b95088b98c190dce3f5061e967b65627ea620bf8e7: Status 404 returned error can't find the container with id 426ce80af7f6e85325d2c1b95088b98c190dce3f5061e967b65627ea620bf8e7
Sep 15 11:59:36.557250 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:36.557158    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d40293-wbs89" event={"ID":"aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15","Type":"ContainerStarted","Data":"dd180783fb78c24cc3fe2aa8fd498e436cf3493e789e05d226b33e4addc5e86b"}
Sep 15 11:59:36.557250 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:36.557211    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d40293-wbs89" event={"ID":"aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15","Type":"ContainerStarted","Data":"426ce80af7f6e85325d2c1b95088b98c190dce3f5061e967b65627ea620bf8e7"}
Sep 15 11:59:36.573958 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:36.573907    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d40293-wbs89" podStartSLOduration=1.573891754 podStartE2EDuration="1.573891754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:59:36.572485847 +0000 UTC m=+2075.234981027" watchObservedRunningTime="2026-09-15 11:59:36.573891754 +0000 UTC m=+2075.236386935"
Sep 15 11:59:37.563467 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:37.563434    2562 generic.go:356] "Generic (PLEG): container finished" podID="aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15" containerID="dd180783fb78c24cc3fe2aa8fd498e436cf3493e789e05d226b33e4addc5e86b" exitCode=0
Sep 15 11:59:37.563875 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:37.563493    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d40293-wbs89" event={"ID":"aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15","Type":"ContainerDied","Data":"dd180783fb78c24cc3fe2aa8fd498e436cf3493e789e05d226b33e4addc5e86b"}
Sep 15 11:59:38.698081 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:38.698043    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d40293-wbs89"
Sep 15 11:59:38.869690 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:38.869664    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15-maas-api-service-ca\") pod \"aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15\" (UID: \"aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15\") "
Sep 15 11:59:38.870001 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:38.869978    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15" (UID: "aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:59:38.970619 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:38.970587    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15-maas-api-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:59:39.574598 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:39.574567    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d40293-wbs89"
Sep 15 11:59:39.574780 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:39.574595    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d40293-wbs89" event={"ID":"aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15","Type":"ContainerDied","Data":"426ce80af7f6e85325d2c1b95088b98c190dce3f5061e967b65627ea620bf8e7"}
Sep 15 11:59:39.574780 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:39.574650    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="426ce80af7f6e85325d2c1b95088b98c190dce3f5061e967b65627ea620bf8e7"
Sep 15 11:59:45.981844 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:45.981808    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"]
Sep 15 11:59:45.982348 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:45.982012    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284" podUID="6c760383-ae98-4d12-814f-04e557d4993a" containerName="maas-api" containerID="cri-o://202dd076e0a870732cb71dabcef17401ca80d212fa511ba778732fac3f85aec9" gracePeriod=30
Sep 15 11:59:46.045171 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.045138    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv"]
Sep 15 11:59:46.045426 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.045382    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv" podUID="1b0fe84c-9592-4b04-a7df-572f46552908" containerName="payload-pre-processing" containerID="cri-o://70d4d9c6b1a0f1094409102473cde465798a91a2857c83bfaa66c42b2ef3812b" gracePeriod=30
Sep 15 11:59:46.196408 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.196371    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d40293-wbs89"]
Sep 15 11:59:46.198572 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.198546    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d40293-wbs89"]
Sep 15 11:59:46.285505 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.285476    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"
Sep 15 11:59:46.305109 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.305087    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv"
Sep 15 11:59:46.428661 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.428628    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c760383-ae98-4d12-814f-04e557d4993a-maas-api-tls\") pod \"6c760383-ae98-4d12-814f-04e557d4993a\" (UID: \"6c760383-ae98-4d12-814f-04e557d4993a\") "
Sep 15 11:59:46.428830 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.428684    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b0fe84c-9592-4b04-a7df-572f46552908-plugins-config-volume\") pod \"1b0fe84c-9592-4b04-a7df-572f46552908\" (UID: \"1b0fe84c-9592-4b04-a7df-572f46552908\") "
Sep 15 11:59:46.428830 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.428711    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c760383-ae98-4d12-814f-04e557d4993a-openshift-service-ca\") pod \"6c760383-ae98-4d12-814f-04e557d4993a\" (UID: \"6c760383-ae98-4d12-814f-04e557d4993a\") "
Sep 15 11:59:46.428830 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.428758    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c760383-ae98-4d12-814f-04e557d4993a-metrics-certs\") pod \"6c760383-ae98-4d12-814f-04e557d4993a\" (UID: \"6c760383-ae98-4d12-814f-04e557d4993a\") "
Sep 15 11:59:46.428830 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.428797    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvt8t\" (UniqueName: \"kubernetes.io/projected/6c760383-ae98-4d12-814f-04e557d4993a-kube-api-access-qvt8t\") pod \"6c760383-ae98-4d12-814f-04e557d4993a\" (UID: \"6c760383-ae98-4d12-814f-04e557d4993a\") "
Sep 15 11:59:46.429091 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.428849    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wrs2\" (UniqueName: \"kubernetes.io/projected/1b0fe84c-9592-4b04-a7df-572f46552908-kube-api-access-4wrs2\") pod \"1b0fe84c-9592-4b04-a7df-572f46552908\" (UID: \"1b0fe84c-9592-4b04-a7df-572f46552908\") "
Sep 15 11:59:46.429157 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.429101    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0fe84c-9592-4b04-a7df-572f46552908-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1b0fe84c-9592-4b04-a7df-572f46552908" (UID: "1b0fe84c-9592-4b04-a7df-572f46552908"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:59:46.429233 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.429161    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b0fe84c-9592-4b04-a7df-572f46552908-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:59:46.429233 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.429156    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c760383-ae98-4d12-814f-04e557d4993a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c760383-ae98-4d12-814f-04e557d4993a" (UID: "6c760383-ae98-4d12-814f-04e557d4993a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:59:46.430864 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.430835    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c760383-ae98-4d12-814f-04e557d4993a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c760383-ae98-4d12-814f-04e557d4993a" (UID: "6c760383-ae98-4d12-814f-04e557d4993a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:59:46.430970 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.430880    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c760383-ae98-4d12-814f-04e557d4993a-kube-api-access-qvt8t" (OuterVolumeSpecName: "kube-api-access-qvt8t") pod "6c760383-ae98-4d12-814f-04e557d4993a" (UID: "6c760383-ae98-4d12-814f-04e557d4993a"). InnerVolumeSpecName "kube-api-access-qvt8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:59:46.430970 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.430885    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0fe84c-9592-4b04-a7df-572f46552908-kube-api-access-4wrs2" (OuterVolumeSpecName: "kube-api-access-4wrs2") pod "1b0fe84c-9592-4b04-a7df-572f46552908" (UID: "1b0fe84c-9592-4b04-a7df-572f46552908"). InnerVolumeSpecName "kube-api-access-4wrs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:59:46.430970 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.430924    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c760383-ae98-4d12-814f-04e557d4993a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6c760383-ae98-4d12-814f-04e557d4993a" (UID: "6c760383-ae98-4d12-814f-04e557d4993a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:59:46.530414 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.530341    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wrs2\" (UniqueName: \"kubernetes.io/projected/1b0fe84c-9592-4b04-a7df-572f46552908-kube-api-access-4wrs2\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:59:46.530414 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.530370    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c760383-ae98-4d12-814f-04e557d4993a-maas-api-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:59:46.530414 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.530382    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c760383-ae98-4d12-814f-04e557d4993a-openshift-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:59:46.530414 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.530393    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c760383-ae98-4d12-814f-04e557d4993a-metrics-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:59:46.530414 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.530405    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvt8t\" (UniqueName: \"kubernetes.io/projected/6c760383-ae98-4d12-814f-04e557d4993a-kube-api-access-qvt8t\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:59:46.608008 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.607975    2562 generic.go:356] "Generic (PLEG): container finished" podID="6c760383-ae98-4d12-814f-04e557d4993a" containerID="202dd076e0a870732cb71dabcef17401ca80d212fa511ba778732fac3f85aec9" exitCode=0
Sep 15 11:59:46.608166 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.608020    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"
Sep 15 11:59:46.608166 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.608074    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284" event={"ID":"6c760383-ae98-4d12-814f-04e557d4993a","Type":"ContainerDied","Data":"202dd076e0a870732cb71dabcef17401ca80d212fa511ba778732fac3f85aec9"}
Sep 15 11:59:46.608166 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.608124    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284" event={"ID":"6c760383-ae98-4d12-814f-04e557d4993a","Type":"ContainerDied","Data":"d09487aebcbbc60d635868afe39c26def4bebbb07831a39adebd7325363db5b2"}
Sep 15 11:59:46.608166 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.608147    2562 scope.go:117] "RemoveContainer" containerID="202dd076e0a870732cb71dabcef17401ca80d212fa511ba778732fac3f85aec9"
Sep 15 11:59:46.609509 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.609489    2562 generic.go:356] "Generic (PLEG): container finished" podID="1b0fe84c-9592-4b04-a7df-572f46552908" containerID="70d4d9c6b1a0f1094409102473cde465798a91a2857c83bfaa66c42b2ef3812b" exitCode=0
Sep 15 11:59:46.609604 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.609548    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv" event={"ID":"1b0fe84c-9592-4b04-a7df-572f46552908","Type":"ContainerDied","Data":"70d4d9c6b1a0f1094409102473cde465798a91a2857c83bfaa66c42b2ef3812b"}
Sep 15 11:59:46.609604 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.609578    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv" event={"ID":"1b0fe84c-9592-4b04-a7df-572f46552908","Type":"ContainerDied","Data":"7243c2dcc460aafa8903ba07fc978f34115a04feb6a46ec63ccacce39df61a1e"}
Sep 15 11:59:46.609604 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.609548    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv"
Sep 15 11:59:46.619494 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.619471    2562 scope.go:117] "RemoveContainer" containerID="202dd076e0a870732cb71dabcef17401ca80d212fa511ba778732fac3f85aec9"
Sep 15 11:59:46.619805 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:59:46.619778    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"202dd076e0a870732cb71dabcef17401ca80d212fa511ba778732fac3f85aec9\": container with ID starting with 202dd076e0a870732cb71dabcef17401ca80d212fa511ba778732fac3f85aec9 not found: ID does not exist" containerID="202dd076e0a870732cb71dabcef17401ca80d212fa511ba778732fac3f85aec9"
Sep 15 11:59:46.619880 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.619816    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"202dd076e0a870732cb71dabcef17401ca80d212fa511ba778732fac3f85aec9"} err="failed to get container status \"202dd076e0a870732cb71dabcef17401ca80d212fa511ba778732fac3f85aec9\": rpc error: code = NotFound desc = could not find container \"202dd076e0a870732cb71dabcef17401ca80d212fa511ba778732fac3f85aec9\": container with ID starting with 202dd076e0a870732cb71dabcef17401ca80d212fa511ba778732fac3f85aec9 not found: ID does not exist"
Sep 15 11:59:46.619880 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.619837    2562 scope.go:117] "RemoveContainer" containerID="70d4d9c6b1a0f1094409102473cde465798a91a2857c83bfaa66c42b2ef3812b"
Sep 15 11:59:46.627343 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.627297    2562 scope.go:117] "RemoveContainer" containerID="70d4d9c6b1a0f1094409102473cde465798a91a2857c83bfaa66c42b2ef3812b"
Sep 15 11:59:46.627905 ip-10-0-135-143 kubenswrapper[2562]: E0915 11:59:46.627880    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d4d9c6b1a0f1094409102473cde465798a91a2857c83bfaa66c42b2ef3812b\": container with ID starting with 70d4d9c6b1a0f1094409102473cde465798a91a2857c83bfaa66c42b2ef3812b not found: ID does not exist" containerID="70d4d9c6b1a0f1094409102473cde465798a91a2857c83bfaa66c42b2ef3812b"
Sep 15 11:59:46.627977 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.627912    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d4d9c6b1a0f1094409102473cde465798a91a2857c83bfaa66c42b2ef3812b"} err="failed to get container status \"70d4d9c6b1a0f1094409102473cde465798a91a2857c83bfaa66c42b2ef3812b\": rpc error: code = NotFound desc = could not find container \"70d4d9c6b1a0f1094409102473cde465798a91a2857c83bfaa66c42b2ef3812b\": container with ID starting with 70d4d9c6b1a0f1094409102473cde465798a91a2857c83bfaa66c42b2ef3812b not found: ID does not exist"
Sep 15 11:59:46.632565 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.632546    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"]
Sep 15 11:59:46.638886 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.638866    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d40293-6b776bbfff-jf284"]
Sep 15 11:59:46.649656 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.649636    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv"]
Sep 15 11:59:46.653199 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:46.653179    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d40293-665f49f49d-xj6pv"]
Sep 15 11:59:47.873940 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:47.873899    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b0fe84c-9592-4b04-a7df-572f46552908" path="/var/lib/kubelet/pods/1b0fe84c-9592-4b04-a7df-572f46552908/volumes"
Sep 15 11:59:47.874481 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:47.874461    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c760383-ae98-4d12-814f-04e557d4993a" path="/var/lib/kubelet/pods/6c760383-ae98-4d12-814f-04e557d4993a/volumes"
Sep 15 11:59:47.874794 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:47.874775    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15" path="/var/lib/kubelet/pods/aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15/volumes"
Sep 15 11:59:49.440019 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:49.439983    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 11:59:53.081216 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:53.081179    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7efa8b-vfp8p"]
Sep 15 11:59:53.081599 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:53.081500    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c760383-ae98-4d12-814f-04e557d4993a" containerName="maas-api"
Sep 15 11:59:53.081599 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:53.081511    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c760383-ae98-4d12-814f-04e557d4993a" containerName="maas-api"
Sep 15 11:59:53.081599 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:53.081529    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15" containerName="revoke-keys"
Sep 15 11:59:53.081599 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:53.081535    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15" containerName="revoke-keys"
Sep 15 11:59:53.081599 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:53.081543    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b0fe84c-9592-4b04-a7df-572f46552908" containerName="payload-pre-processing"
Sep 15 11:59:53.081599 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:53.081548    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b0fe84c-9592-4b04-a7df-572f46552908" containerName="payload-pre-processing"
Sep 15 11:59:53.081599 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:53.081597    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b0fe84c-9592-4b04-a7df-572f46552908" containerName="payload-pre-processing"
Sep 15 11:59:53.081820 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:53.081604    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c760383-ae98-4d12-814f-04e557d4993a" containerName="maas-api"
Sep 15 11:59:53.081820 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:53.081613    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="aefe90bc-ff92-43d9-a9c5-bac8fcfeaf15" containerName="revoke-keys"
Sep 15 11:59:53.086023 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:53.086004    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7efa8b-vfp8p"
Sep 15 11:59:53.088129 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:53.088105    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:59:53.088248 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:53.088105    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rbxjk\""
Sep 15 11:59:53.092421 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:53.092398    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7efa8b-vfp8p"]
Sep 15 11:59:53.184545 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:53.184511    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a94892-4fc9-478b-86b4-972dd88dab0c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7efa8b-vfp8p\" (UID: \"d0a94892-4fc9-478b-86b4-972dd88dab0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7efa8b-vfp8p"
Sep 15 11:59:53.285408 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:53.285377    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a94892-4fc9-478b-86b4-972dd88dab0c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7efa8b-vfp8p\" (UID: \"d0a94892-4fc9-478b-86b4-972dd88dab0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7efa8b-vfp8p"
Sep 15 11:59:53.285975 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:53.285956    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a94892-4fc9-478b-86b4-972dd88dab0c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7efa8b-vfp8p\" (UID: \"d0a94892-4fc9-478b-86b4-972dd88dab0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7efa8b-vfp8p"
Sep 15 11:59:53.396568 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:53.396491    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7efa8b-vfp8p"
Sep 15 11:59:53.668736 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:53.668710    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7efa8b-vfp8p"]
Sep 15 11:59:53.671051 ip-10-0-135-143 kubenswrapper[2562]: W0915 11:59:53.671020    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a94892_4fc9_478b_86b4_972dd88dab0c.slice/crio-311e89d492579295bc904ab7b0423ae8d4565a6a26df0d2ae88399608ca21c97 WatchSource:0}: Error finding container 311e89d492579295bc904ab7b0423ae8d4565a6a26df0d2ae88399608ca21c97: Status 404 returned error can't find the container with id 311e89d492579295bc904ab7b0423ae8d4565a6a26df0d2ae88399608ca21c97
Sep 15 11:59:54.647949 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:54.647919    2562 generic.go:356] "Generic (PLEG): container finished" podID="d0a94892-4fc9-478b-86b4-972dd88dab0c" containerID="910965b72c06842d532f50893af33cb722e224368ad2eee53eaf1b27f12bb5f1" exitCode=0
Sep 15 11:59:54.648380 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:54.648004    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7efa8b-vfp8p" event={"ID":"d0a94892-4fc9-478b-86b4-972dd88dab0c","Type":"ContainerDied","Data":"910965b72c06842d532f50893af33cb722e224368ad2eee53eaf1b27f12bb5f1"}
Sep 15 11:59:54.648380 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:54.648066    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7efa8b-vfp8p" event={"ID":"d0a94892-4fc9-478b-86b4-972dd88dab0c","Type":"ContainerStarted","Data":"311e89d492579295bc904ab7b0423ae8d4565a6a26df0d2ae88399608ca21c97"}
Sep 15 11:59:55.777047 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:55.777022    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7efa8b-vfp8p"
Sep 15 11:59:55.808548 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:55.808516    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a94892-4fc9-478b-86b4-972dd88dab0c-maas-api-service-ca\") pod \"d0a94892-4fc9-478b-86b4-972dd88dab0c\" (UID: \"d0a94892-4fc9-478b-86b4-972dd88dab0c\") "
Sep 15 11:59:55.808895 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:55.808875    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a94892-4fc9-478b-86b4-972dd88dab0c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d0a94892-4fc9-478b-86b4-972dd88dab0c" (UID: "d0a94892-4fc9-478b-86b4-972dd88dab0c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:59:55.909234 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:55.909163    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a94892-4fc9-478b-86b4-972dd88dab0c-maas-api-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 11:59:56.658656 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:56.658620    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7efa8b-vfp8p"
Sep 15 11:59:56.658656 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:56.658647    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7efa8b-vfp8p" event={"ID":"d0a94892-4fc9-478b-86b4-972dd88dab0c","Type":"ContainerDied","Data":"311e89d492579295bc904ab7b0423ae8d4565a6a26df0d2ae88399608ca21c97"}
Sep 15 11:59:56.658919 ip-10-0-135-143 kubenswrapper[2562]: I0915 11:59:56.658693    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="311e89d492579295bc904ab7b0423ae8d4565a6a26df0d2ae88399608ca21c97"
Sep 15 12:00:00.130100 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:00.130068    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824560-k4v2h"]
Sep 15 12:00:00.130527 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:00.130509    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a94892-4fc9-478b-86b4-972dd88dab0c" containerName="revoke-keys"
Sep 15 12:00:00.130576 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:00.130531    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a94892-4fc9-478b-86b4-972dd88dab0c" containerName="revoke-keys"
Sep 15 12:00:00.130655 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:00.130644    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a94892-4fc9-478b-86b4-972dd88dab0c" containerName="revoke-keys"
Sep 15 12:00:00.135220 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:00.135193    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7efa8b-29824560-r7vx8"]
Sep 15 12:00:00.135373 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:00.135297    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824560-k4v2h"
Sep 15 12:00:00.138033 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:00.138001    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rbxjk\""
Sep 15 12:00:00.139632 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:00.139614    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7efa8b-29824560-r7vx8"
Sep 15 12:00:00.145821 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:00.145752    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824560-k4v2h"]
Sep 15 12:00:00.146719 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:00.146696    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824560-k4v2h"
Sep 15 12:00:00.151462 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:00.151441    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7efa8b-29824560-r7vx8"]
Sep 15 12:00:00.155155 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:00.155135    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7efa8b-29824560-r7vx8"
Sep 15 12:00:00.427103 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:00.427078    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824560-k4v2h"]
Sep 15 12:00:00.429993 ip-10-0-135-143 kubenswrapper[2562]: W0915 12:00:00.429963    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a51cee_66e1_4e3c_ae56_3223ab47c915.slice/crio-3724ace93ab59142073b73be3096dc7f801a53feaff59f3d7c997ab4bc74fc4c WatchSource:0}: Error finding container 3724ace93ab59142073b73be3096dc7f801a53feaff59f3d7c997ab4bc74fc4c: Status 404 returned error can't find the container with id 3724ace93ab59142073b73be3096dc7f801a53feaff59f3d7c997ab4bc74fc4c
Sep 15 12:00:00.432754 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:00.432596    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7efa8b-29824560-r7vx8"]
Sep 15 12:00:00.435110 ip-10-0-135-143 kubenswrapper[2562]: W0915 12:00:00.435088    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d8edde_fa9f_4037_96d7_09eddf383b10.slice/crio-6508756986d9a755f5af0551dc7daea1f1ebf78e6332b7535490227243bc560b WatchSource:0}: Error finding container 6508756986d9a755f5af0551dc7daea1f1ebf78e6332b7535490227243bc560b: Status 404 returned error can't find the container with id 6508756986d9a755f5af0551dc7daea1f1ebf78e6332b7535490227243bc560b
Sep 15 12:00:00.676841 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:00.676808    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7efa8b-29824560-r7vx8" event={"ID":"71d8edde-fa9f-4037-96d7-09eddf383b10","Type":"ContainerStarted","Data":"6508756986d9a755f5af0551dc7daea1f1ebf78e6332b7535490227243bc560b"}
Sep 15 12:00:00.678043 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:00.677993    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824560-k4v2h" event={"ID":"94a51cee-66e1-4e3c-ae56-3223ab47c915","Type":"ContainerStarted","Data":"3724ace93ab59142073b73be3096dc7f801a53feaff59f3d7c997ab4bc74fc4c"}
Sep 15 12:00:01.684468 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:01.684431    2562 generic.go:356] "Generic (PLEG): container finished" podID="94a51cee-66e1-4e3c-ae56-3223ab47c915" containerID="65b46cb9c85991bc58dbedf0e36a8b5c275f0276ea49c93c45db68453b37c628" exitCode=0
Sep 15 12:00:01.684910 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:01.684521    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824560-k4v2h" event={"ID":"94a51cee-66e1-4e3c-ae56-3223ab47c915","Type":"ContainerDied","Data":"65b46cb9c85991bc58dbedf0e36a8b5c275f0276ea49c93c45db68453b37c628"}
Sep 15 12:00:01.686163 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:01.686140    2562 generic.go:356] "Generic (PLEG): container finished" podID="71d8edde-fa9f-4037-96d7-09eddf383b10" containerID="f27f49baecbcea8b93dae50b638f462532d64598e8d8f27218f4f217741107f8" exitCode=0
Sep 15 12:00:01.686264 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:01.686192    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7efa8b-29824560-r7vx8" event={"ID":"71d8edde-fa9f-4037-96d7-09eddf383b10","Type":"ContainerDied","Data":"f27f49baecbcea8b93dae50b638f462532d64598e8d8f27218f4f217741107f8"}
Sep 15 12:00:02.827710 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:02.827674    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824560-k4v2h"
Sep 15 12:00:02.853640 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:02.853616    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7efa8b-29824560-r7vx8"
Sep 15 12:00:03.691639 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:03.691598    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7efa8b-29824560-r7vx8"]
Sep 15 12:00:03.697363 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:03.697340    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7efa8b-29824560-r7vx8"
Sep 15 12:00:03.697501 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:03.697384    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7efa8b-29824560-r7vx8" event={"ID":"71d8edde-fa9f-4037-96d7-09eddf383b10","Type":"ContainerDied","Data":"6508756986d9a755f5af0551dc7daea1f1ebf78e6332b7535490227243bc560b"}
Sep 15 12:00:03.697501 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:03.697428    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6508756986d9a755f5af0551dc7daea1f1ebf78e6332b7535490227243bc560b"
Sep 15 12:00:03.699034 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:03.699011    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7efa8b-29824560-r7vx8"]
Sep 15 12:00:03.699187 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:03.699154    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824560-k4v2h"
Sep 15 12:00:03.699256 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:03.699180    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824560-k4v2h" event={"ID":"94a51cee-66e1-4e3c-ae56-3223ab47c915","Type":"ContainerDied","Data":"3724ace93ab59142073b73be3096dc7f801a53feaff59f3d7c997ab4bc74fc4c"}
Sep 15 12:00:03.699256 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:03.699213    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3724ace93ab59142073b73be3096dc7f801a53feaff59f3d7c997ab4bc74fc4c"
Sep 15 12:00:03.708875 ip-10-0-135-143 kubenswrapper[2562]: E0915 12:00:03.708842    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-shared-a-w3-7efa8b-29824560-r7vx8\" is forbidden: User \"system:node:ip-10-0-135-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-135-143.ec2.internal' and this object" podUID="71d8edde-fa9f-4037-96d7-09eddf383b10" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7efa8b-29824560-r7vx8"
Sep 15 12:00:03.711771 ip-10-0-135-143 kubenswrapper[2562]: E0915 12:00:03.711741    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-shared-a-w3-7efa8b-29824560-r7vx8\" is forbidden: User \"system:node:ip-10-0-135-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-135-143.ec2.internal' and this object" podUID="71d8edde-fa9f-4037-96d7-09eddf383b10" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7efa8b-29824560-r7vx8"
Sep 15 12:00:03.730011 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:03.729977    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5"]
Sep 15 12:00:03.730344 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:03.730314    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5" podUID="b7e1eca0-7a52-476e-9f5f-5ee74dcd7113" containerName="payload-pre-processing" containerID="cri-o://b64e4b19e817f6f30dbd4faa013f9907be9cce07d98b7c7e9d00af9339ae6bda" gracePeriod=30
Sep 15 12:00:03.732774 ip-10-0-135-143 kubenswrapper[2562]: E0915 12:00:03.732736    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-shared-a-w3-7efa8b-29824560-r7vx8\" is forbidden: User \"system:node:ip-10-0-135-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-135-143.ec2.internal' and this object" podUID="71d8edde-fa9f-4037-96d7-09eddf383b10" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7efa8b-29824560-r7vx8"
Sep 15 12:00:03.874695 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:03.874663    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71d8edde-fa9f-4037-96d7-09eddf383b10" path="/var/lib/kubelet/pods/71d8edde-fa9f-4037-96d7-09eddf383b10/volumes"
Sep 15 12:00:03.930992 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:03.930961    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7efa8b-vfp8p"]
Sep 15 12:00:03.936454 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:03.936424    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7efa8b-vfp8p"]
Sep 15 12:00:03.989242 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:03.989218    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5"
Sep 15 12:00:04.081225 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:04.081192    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7e1eca0-7a52-476e-9f5f-5ee74dcd7113-plugins-config-volume\") pod \"b7e1eca0-7a52-476e-9f5f-5ee74dcd7113\" (UID: \"b7e1eca0-7a52-476e-9f5f-5ee74dcd7113\") "
Sep 15 12:00:04.081398 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:04.081261    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqbsp\" (UniqueName: \"kubernetes.io/projected/b7e1eca0-7a52-476e-9f5f-5ee74dcd7113-kube-api-access-mqbsp\") pod \"b7e1eca0-7a52-476e-9f5f-5ee74dcd7113\" (UID: \"b7e1eca0-7a52-476e-9f5f-5ee74dcd7113\") "
Sep 15 12:00:04.081643 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:04.081619    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7e1eca0-7a52-476e-9f5f-5ee74dcd7113-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7e1eca0-7a52-476e-9f5f-5ee74dcd7113" (UID: "b7e1eca0-7a52-476e-9f5f-5ee74dcd7113"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:00:04.083375 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:04.083356    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7e1eca0-7a52-476e-9f5f-5ee74dcd7113-kube-api-access-mqbsp" (OuterVolumeSpecName: "kube-api-access-mqbsp") pod "b7e1eca0-7a52-476e-9f5f-5ee74dcd7113" (UID: "b7e1eca0-7a52-476e-9f5f-5ee74dcd7113"). InnerVolumeSpecName "kube-api-access-mqbsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:00:04.182708 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:04.182676    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7e1eca0-7a52-476e-9f5f-5ee74dcd7113-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 12:00:04.182708 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:04.182704    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqbsp\" (UniqueName: \"kubernetes.io/projected/b7e1eca0-7a52-476e-9f5f-5ee74dcd7113-kube-api-access-mqbsp\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 12:00:04.708604 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:04.708566    2562 generic.go:356] "Generic (PLEG): container finished" podID="b7e1eca0-7a52-476e-9f5f-5ee74dcd7113" containerID="b64e4b19e817f6f30dbd4faa013f9907be9cce07d98b7c7e9d00af9339ae6bda" exitCode=0
Sep 15 12:00:04.708797 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:04.708618    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5"
Sep 15 12:00:04.708797 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:04.708657    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5" event={"ID":"b7e1eca0-7a52-476e-9f5f-5ee74dcd7113","Type":"ContainerDied","Data":"b64e4b19e817f6f30dbd4faa013f9907be9cce07d98b7c7e9d00af9339ae6bda"}
Sep 15 12:00:04.708797 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:04.708718    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5" event={"ID":"b7e1eca0-7a52-476e-9f5f-5ee74dcd7113","Type":"ContainerDied","Data":"647b597bb22bc3860a4e709141b6f5f9524db56e3361fc93d5e73bdb778663c5"}
Sep 15 12:00:04.708797 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:04.708742    2562 scope.go:117] "RemoveContainer" containerID="b64e4b19e817f6f30dbd4faa013f9907be9cce07d98b7c7e9d00af9339ae6bda"
Sep 15 12:00:04.718025 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:04.718008    2562 scope.go:117] "RemoveContainer" containerID="b64e4b19e817f6f30dbd4faa013f9907be9cce07d98b7c7e9d00af9339ae6bda"
Sep 15 12:00:04.718332 ip-10-0-135-143 kubenswrapper[2562]: E0915 12:00:04.718307    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b64e4b19e817f6f30dbd4faa013f9907be9cce07d98b7c7e9d00af9339ae6bda\": container with ID starting with b64e4b19e817f6f30dbd4faa013f9907be9cce07d98b7c7e9d00af9339ae6bda not found: ID does not exist" containerID="b64e4b19e817f6f30dbd4faa013f9907be9cce07d98b7c7e9d00af9339ae6bda"
Sep 15 12:00:04.718412 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:04.718342    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b64e4b19e817f6f30dbd4faa013f9907be9cce07d98b7c7e9d00af9339ae6bda"} err="failed to get container status \"b64e4b19e817f6f30dbd4faa013f9907be9cce07d98b7c7e9d00af9339ae6bda\": rpc error: code = NotFound desc = could not find container \"b64e4b19e817f6f30dbd4faa013f9907be9cce07d98b7c7e9d00af9339ae6bda\": container with ID starting with b64e4b19e817f6f30dbd4faa013f9907be9cce07d98b7c7e9d00af9339ae6bda not found: ID does not exist"
Sep 15 12:00:04.734188 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:04.734162    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5"]
Sep 15 12:00:04.740399 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:04.740379    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7efa8b-545cff7746-twdn5"]
Sep 15 12:00:05.873405 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:05.873373    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7e1eca0-7a52-476e-9f5f-5ee74dcd7113" path="/var/lib/kubelet/pods/b7e1eca0-7a52-476e-9f5f-5ee74dcd7113/volumes"
Sep 15 12:00:05.873812 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:05.873696    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0a94892-4fc9-478b-86b4-972dd88dab0c" path="/var/lib/kubelet/pods/d0a94892-4fc9-478b-86b4-972dd88dab0c/volumes"
Sep 15 12:00:06.790621 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:06.790587    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:00:10.782915 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:10.782884    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2cc790-zpg9q"]
Sep 15 12:00:10.783333 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:10.783209    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71d8edde-fa9f-4037-96d7-09eddf383b10" containerName="cleanup"
Sep 15 12:00:10.783333 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:10.783222    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="71d8edde-fa9f-4037-96d7-09eddf383b10" containerName="cleanup"
Sep 15 12:00:10.783333 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:10.783233    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7e1eca0-7a52-476e-9f5f-5ee74dcd7113" containerName="payload-pre-processing"
Sep 15 12:00:10.783333 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:10.783239    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7e1eca0-7a52-476e-9f5f-5ee74dcd7113" containerName="payload-pre-processing"
Sep 15 12:00:10.783333 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:10.783247    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94a51cee-66e1-4e3c-ae56-3223ab47c915" containerName="cleanup"
Sep 15 12:00:10.783333 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:10.783252    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="94a51cee-66e1-4e3c-ae56-3223ab47c915" containerName="cleanup"
Sep 15 12:00:10.783333 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:10.783312    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="94a51cee-66e1-4e3c-ae56-3223ab47c915" containerName="cleanup"
Sep 15 12:00:10.783333 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:10.783321    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="71d8edde-fa9f-4037-96d7-09eddf383b10" containerName="cleanup"
Sep 15 12:00:10.783333 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:10.783328    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7e1eca0-7a52-476e-9f5f-5ee74dcd7113" containerName="payload-pre-processing"
Sep 15 12:00:10.786632 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:10.786611    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2cc790-zpg9q"
Sep 15 12:00:10.788705 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:10.788685    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:00:10.788828 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:10.788751    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rbxjk\""
Sep 15 12:00:10.793864 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:10.793842    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2cc790-zpg9q"]
Sep 15 12:00:10.939280 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:10.939246    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/decbf4cf-f6c9-4b0b-abb4-a555ae357b95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2cc790-zpg9q\" (UID: \"decbf4cf-f6c9-4b0b-abb4-a555ae357b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2cc790-zpg9q"
Sep 15 12:00:11.040470 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:11.040387    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/decbf4cf-f6c9-4b0b-abb4-a555ae357b95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2cc790-zpg9q\" (UID: \"decbf4cf-f6c9-4b0b-abb4-a555ae357b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2cc790-zpg9q"
Sep 15 12:00:11.041011 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:11.040993    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/decbf4cf-f6c9-4b0b-abb4-a555ae357b95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2cc790-zpg9q\" (UID: \"decbf4cf-f6c9-4b0b-abb4-a555ae357b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2cc790-zpg9q"
Sep 15 12:00:11.098078 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:11.098024    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2cc790-zpg9q"
Sep 15 12:00:11.371815 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:11.371791    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2cc790-zpg9q"]
Sep 15 12:00:11.374375 ip-10-0-135-143 kubenswrapper[2562]: W0915 12:00:11.374340    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddecbf4cf_f6c9_4b0b_abb4_a555ae357b95.slice/crio-5069cf87d8a9ad6b41a6defc0d0b4ef47b9b9745bf4c5f3d7e9c29cbb5913b95 WatchSource:0}: Error finding container 5069cf87d8a9ad6b41a6defc0d0b4ef47b9b9745bf4c5f3d7e9c29cbb5913b95: Status 404 returned error can't find the container with id 5069cf87d8a9ad6b41a6defc0d0b4ef47b9b9745bf4c5f3d7e9c29cbb5913b95
Sep 15 12:00:11.743051 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:11.742966    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2cc790-zpg9q" event={"ID":"decbf4cf-f6c9-4b0b-abb4-a555ae357b95","Type":"ContainerStarted","Data":"f289211ca8a010eb8a3ec2be0abdb95df7c97859fe74d2ef1030e6a4d8f0362c"}
Sep 15 12:00:11.743051 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:11.743018    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2cc790-zpg9q" event={"ID":"decbf4cf-f6c9-4b0b-abb4-a555ae357b95","Type":"ContainerStarted","Data":"5069cf87d8a9ad6b41a6defc0d0b4ef47b9b9745bf4c5f3d7e9c29cbb5913b95"}
Sep 15 12:00:11.757639 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:11.757595    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2cc790-zpg9q" podStartSLOduration=1.757580436 podStartE2EDuration="1.757580436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:00:11.756858848 +0000 UTC m=+2110.419354063" watchObservedRunningTime="2026-09-15 12:00:11.757580436 +0000 UTC m=+2110.420075617"
Sep 15 12:00:12.749165 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:12.749132    2562 generic.go:356] "Generic (PLEG): container finished" podID="decbf4cf-f6c9-4b0b-abb4-a555ae357b95" containerID="f289211ca8a010eb8a3ec2be0abdb95df7c97859fe74d2ef1030e6a4d8f0362c" exitCode=0
Sep 15 12:00:12.749539 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:12.749216    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2cc790-zpg9q" event={"ID":"decbf4cf-f6c9-4b0b-abb4-a555ae357b95","Type":"ContainerDied","Data":"f289211ca8a010eb8a3ec2be0abdb95df7c97859fe74d2ef1030e6a4d8f0362c"}
Sep 15 12:00:13.883805 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:13.883781    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2cc790-zpg9q"
Sep 15 12:00:13.966447 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:13.966411    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/decbf4cf-f6c9-4b0b-abb4-a555ae357b95-maas-api-service-ca\") pod \"decbf4cf-f6c9-4b0b-abb4-a555ae357b95\" (UID: \"decbf4cf-f6c9-4b0b-abb4-a555ae357b95\") "
Sep 15 12:00:13.966764 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:13.966740    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/decbf4cf-f6c9-4b0b-abb4-a555ae357b95-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "decbf4cf-f6c9-4b0b-abb4-a555ae357b95" (UID: "decbf4cf-f6c9-4b0b-abb4-a555ae357b95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:00:14.067023 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:14.066932    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/decbf4cf-f6c9-4b0b-abb4-a555ae357b95-maas-api-service-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 12:00:14.759734 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:14.759702    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2cc790-zpg9q"
Sep 15 12:00:14.759734 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:14.759718    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2cc790-zpg9q" event={"ID":"decbf4cf-f6c9-4b0b-abb4-a555ae357b95","Type":"ContainerDied","Data":"5069cf87d8a9ad6b41a6defc0d0b4ef47b9b9745bf4c5f3d7e9c29cbb5913b95"}
Sep 15 12:00:14.759954 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:14.759768    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5069cf87d8a9ad6b41a6defc0d0b4ef47b9b9745bf4c5f3d7e9c29cbb5913b95"
Sep 15 12:00:21.299442 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.299402    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp"]
Sep 15 12:00:21.299902 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.299675    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp" podUID="8080f8e1-0e28-4fef-a1da-9eedd98b5fd0" containerName="payload-processing" containerID="cri-o://6bc35efc27b4902474fb9fd88b8310e567f8398d1b4e92f59dc85a7e09109d2b" gracePeriod=30
Sep 15 12:00:21.316747 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.316717    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6"]
Sep 15 12:00:21.316969 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.316943    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6" podUID="fa6f4f0c-ebed-466b-b6af-19d74818d559" containerName="payload-pre-processing" containerID="cri-o://72d700bd74fe1821a1fd7cc9ef1e775c23dd17ef7b422d0bd4e465b50c1d791c" gracePeriod=30
Sep 15 12:00:21.494863 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.494830    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2cc790-zpg9q"]
Sep 15 12:00:21.502667 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.502638    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2cc790-zpg9q"]
Sep 15 12:00:21.599022 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.598997    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6"
Sep 15 12:00:21.601738 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.601721    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp"
Sep 15 12:00:21.733911 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.733877    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9cwc\" (UniqueName: \"kubernetes.io/projected/8080f8e1-0e28-4fef-a1da-9eedd98b5fd0-kube-api-access-v9cwc\") pod \"8080f8e1-0e28-4fef-a1da-9eedd98b5fd0\" (UID: \"8080f8e1-0e28-4fef-a1da-9eedd98b5fd0\") "
Sep 15 12:00:21.734095 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.733951    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9x2h\" (UniqueName: \"kubernetes.io/projected/fa6f4f0c-ebed-466b-b6af-19d74818d559-kube-api-access-j9x2h\") pod \"fa6f4f0c-ebed-466b-b6af-19d74818d559\" (UID: \"fa6f4f0c-ebed-466b-b6af-19d74818d559\") "
Sep 15 12:00:21.734095 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.734014    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6f4f0c-ebed-466b-b6af-19d74818d559-plugins-config-volume\") pod \"fa6f4f0c-ebed-466b-b6af-19d74818d559\" (UID: \"fa6f4f0c-ebed-466b-b6af-19d74818d559\") "
Sep 15 12:00:21.734095 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.734043    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8080f8e1-0e28-4fef-a1da-9eedd98b5fd0-plugins-config-volume\") pod \"8080f8e1-0e28-4fef-a1da-9eedd98b5fd0\" (UID: \"8080f8e1-0e28-4fef-a1da-9eedd98b5fd0\") "
Sep 15 12:00:21.734413 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.734389    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6f4f0c-ebed-466b-b6af-19d74818d559-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa6f4f0c-ebed-466b-b6af-19d74818d559" (UID: "fa6f4f0c-ebed-466b-b6af-19d74818d559"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:00:21.734484 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.734428    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8080f8e1-0e28-4fef-a1da-9eedd98b5fd0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8080f8e1-0e28-4fef-a1da-9eedd98b5fd0" (UID: "8080f8e1-0e28-4fef-a1da-9eedd98b5fd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:00:21.736130 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.736105    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8080f8e1-0e28-4fef-a1da-9eedd98b5fd0-kube-api-access-v9cwc" (OuterVolumeSpecName: "kube-api-access-v9cwc") pod "8080f8e1-0e28-4fef-a1da-9eedd98b5fd0" (UID: "8080f8e1-0e28-4fef-a1da-9eedd98b5fd0"). InnerVolumeSpecName "kube-api-access-v9cwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:00:21.736130 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.736117    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6f4f0c-ebed-466b-b6af-19d74818d559-kube-api-access-j9x2h" (OuterVolumeSpecName: "kube-api-access-j9x2h") pod "fa6f4f0c-ebed-466b-b6af-19d74818d559" (UID: "fa6f4f0c-ebed-466b-b6af-19d74818d559"). InnerVolumeSpecName "kube-api-access-j9x2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:00:21.796788 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.796750    2562 generic.go:356] "Generic (PLEG): container finished" podID="fa6f4f0c-ebed-466b-b6af-19d74818d559" containerID="72d700bd74fe1821a1fd7cc9ef1e775c23dd17ef7b422d0bd4e465b50c1d791c" exitCode=0
Sep 15 12:00:21.796974 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.796809    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6"
Sep 15 12:00:21.796974 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.796839    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6" event={"ID":"fa6f4f0c-ebed-466b-b6af-19d74818d559","Type":"ContainerDied","Data":"72d700bd74fe1821a1fd7cc9ef1e775c23dd17ef7b422d0bd4e465b50c1d791c"}
Sep 15 12:00:21.796974 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.796928    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6" event={"ID":"fa6f4f0c-ebed-466b-b6af-19d74818d559","Type":"ContainerDied","Data":"17d0bf311a53959280a3637f346a8e7b468bf866b816acf70aea4e725d387335"}
Sep 15 12:00:21.796974 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.796956    2562 scope.go:117] "RemoveContainer" containerID="72d700bd74fe1821a1fd7cc9ef1e775c23dd17ef7b422d0bd4e465b50c1d791c"
Sep 15 12:00:21.798295 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.798229    2562 generic.go:356] "Generic (PLEG): container finished" podID="8080f8e1-0e28-4fef-a1da-9eedd98b5fd0" containerID="6bc35efc27b4902474fb9fd88b8310e567f8398d1b4e92f59dc85a7e09109d2b" exitCode=0
Sep 15 12:00:21.798374 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.798321    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp" event={"ID":"8080f8e1-0e28-4fef-a1da-9eedd98b5fd0","Type":"ContainerDied","Data":"6bc35efc27b4902474fb9fd88b8310e567f8398d1b4e92f59dc85a7e09109d2b"}
Sep 15 12:00:21.798374 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.798351    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp"
Sep 15 12:00:21.798491 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.798354    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp" event={"ID":"8080f8e1-0e28-4fef-a1da-9eedd98b5fd0","Type":"ContainerDied","Data":"237294044bca875a1f86457f9fbaef899e6e04d9462c274d0b992150475ec9c5"}
Sep 15 12:00:21.810162 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.810113    2562 scope.go:117] "RemoveContainer" containerID="72d700bd74fe1821a1fd7cc9ef1e775c23dd17ef7b422d0bd4e465b50c1d791c"
Sep 15 12:00:21.810509 ip-10-0-135-143 kubenswrapper[2562]: E0915 12:00:21.810489    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d700bd74fe1821a1fd7cc9ef1e775c23dd17ef7b422d0bd4e465b50c1d791c\": container with ID starting with 72d700bd74fe1821a1fd7cc9ef1e775c23dd17ef7b422d0bd4e465b50c1d791c not found: ID does not exist" containerID="72d700bd74fe1821a1fd7cc9ef1e775c23dd17ef7b422d0bd4e465b50c1d791c"
Sep 15 12:00:21.810601 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.810520    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d700bd74fe1821a1fd7cc9ef1e775c23dd17ef7b422d0bd4e465b50c1d791c"} err="failed to get container status \"72d700bd74fe1821a1fd7cc9ef1e775c23dd17ef7b422d0bd4e465b50c1d791c\": rpc error: code = NotFound desc = could not find container \"72d700bd74fe1821a1fd7cc9ef1e775c23dd17ef7b422d0bd4e465b50c1d791c\": container with ID starting with 72d700bd74fe1821a1fd7cc9ef1e775c23dd17ef7b422d0bd4e465b50c1d791c not found: ID does not exist"
Sep 15 12:00:21.810601 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.810544    2562 scope.go:117] "RemoveContainer" containerID="6bc35efc27b4902474fb9fd88b8310e567f8398d1b4e92f59dc85a7e09109d2b"
Sep 15 12:00:21.818251 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.818232    2562 scope.go:117] "RemoveContainer" containerID="6bc35efc27b4902474fb9fd88b8310e567f8398d1b4e92f59dc85a7e09109d2b"
Sep 15 12:00:21.818542 ip-10-0-135-143 kubenswrapper[2562]: E0915 12:00:21.818524    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bc35efc27b4902474fb9fd88b8310e567f8398d1b4e92f59dc85a7e09109d2b\": container with ID starting with 6bc35efc27b4902474fb9fd88b8310e567f8398d1b4e92f59dc85a7e09109d2b not found: ID does not exist" containerID="6bc35efc27b4902474fb9fd88b8310e567f8398d1b4e92f59dc85a7e09109d2b"
Sep 15 12:00:21.818641 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.818549    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc35efc27b4902474fb9fd88b8310e567f8398d1b4e92f59dc85a7e09109d2b"} err="failed to get container status \"6bc35efc27b4902474fb9fd88b8310e567f8398d1b4e92f59dc85a7e09109d2b\": rpc error: code = NotFound desc = could not find container \"6bc35efc27b4902474fb9fd88b8310e567f8398d1b4e92f59dc85a7e09109d2b\": container with ID starting with 6bc35efc27b4902474fb9fd88b8310e567f8398d1b4e92f59dc85a7e09109d2b not found: ID does not exist"
Sep 15 12:00:21.825729 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.825706    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6"]
Sep 15 12:00:21.833160 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.833140    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2cc790-65464dff98-nw9r6"]
Sep 15 12:00:21.834718 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.834697    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8080f8e1-0e28-4fef-a1da-9eedd98b5fd0-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 12:00:21.834785 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.834722    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9cwc\" (UniqueName: \"kubernetes.io/projected/8080f8e1-0e28-4fef-a1da-9eedd98b5fd0-kube-api-access-v9cwc\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 12:00:21.834785 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.834738    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9x2h\" (UniqueName: \"kubernetes.io/projected/fa6f4f0c-ebed-466b-b6af-19d74818d559-kube-api-access-j9x2h\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 12:00:21.834785 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.834751    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6f4f0c-ebed-466b-b6af-19d74818d559-plugins-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 12:00:21.842851 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.842827    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp"]
Sep 15 12:00:21.847554 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.847533    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2cc790-597c44d749-gqvdp"]
Sep 15 12:00:21.872720 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.872694    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8080f8e1-0e28-4fef-a1da-9eedd98b5fd0" path="/var/lib/kubelet/pods/8080f8e1-0e28-4fef-a1da-9eedd98b5fd0/volumes"
Sep 15 12:00:21.873017 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.873005    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="decbf4cf-f6c9-4b0b-abb4-a555ae357b95" path="/var/lib/kubelet/pods/decbf4cf-f6c9-4b0b-abb4-a555ae357b95/volumes"
Sep 15 12:00:21.873330 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:21.873318    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa6f4f0c-ebed-466b-b6af-19d74818d559" path="/var/lib/kubelet/pods/fa6f4f0c-ebed-466b-b6af-19d74818d559/volumes"
Sep 15 12:00:23.860151 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:23.860115    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:00:32.649072 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:00:32.649017    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:01:53.143107 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:01:53.143073    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:02:19.339814 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:02:19.339779    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:02:27.241791 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:02:27.241718    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:02:35.150622 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:02:35.150582    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:02:51.148711 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:02:51.148678    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:02:59.739001 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:02:59.738966    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:04:25.350310 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:04:25.350274    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:04:33.653472 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:04:33.653431    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:04:41.052788 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:04:41.052746    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:05:00.552768 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.552735    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"]
Sep 15 12:05:00.553234 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.553067    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8080f8e1-0e28-4fef-a1da-9eedd98b5fd0" containerName="payload-processing"
Sep 15 12:05:00.553234 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.553081    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8080f8e1-0e28-4fef-a1da-9eedd98b5fd0" containerName="payload-processing"
Sep 15 12:05:00.553234 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.553091    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa6f4f0c-ebed-466b-b6af-19d74818d559" containerName="payload-pre-processing"
Sep 15 12:05:00.553234 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.553097    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa6f4f0c-ebed-466b-b6af-19d74818d559" containerName="payload-pre-processing"
Sep 15 12:05:00.553234 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.553113    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="decbf4cf-f6c9-4b0b-abb4-a555ae357b95" containerName="revoke-keys"
Sep 15 12:05:00.553234 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.553122    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="decbf4cf-f6c9-4b0b-abb4-a555ae357b95" containerName="revoke-keys"
Sep 15 12:05:00.553234 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.553168    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa6f4f0c-ebed-466b-b6af-19d74818d559" containerName="payload-pre-processing"
Sep 15 12:05:00.553234 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.553178    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8080f8e1-0e28-4fef-a1da-9eedd98b5fd0" containerName="payload-processing"
Sep 15 12:05:00.553234 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.553184    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="decbf4cf-f6c9-4b0b-abb4-a555ae357b95" containerName="revoke-keys"
Sep 15 12:05:00.556159 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.556142    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:00.558383 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.558362    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-a80b3a05-kserve-self-signed-certs\""
Sep 15 12:05:00.566453 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.566430    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"]
Sep 15 12:05:00.617908 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.617884    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-home\") pod \"e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") " pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:00.618082 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.617930    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-tmp-dir\") pod \"e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") " pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:00.618082 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.617951    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-dshm\") pod \"e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") " pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:00.618082 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.618020    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-model-cache\") pod \"e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") " pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:00.618082 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.618076    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4j6m\" (UniqueName: \"kubernetes.io/projected/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-kube-api-access-r4j6m\") pod \"e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") " pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:00.618253 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.618126    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-tls-certs\") pod \"e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") " pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:00.719249 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.719216    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-tls-certs\") pod \"e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") " pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:00.719404 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.719261    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-home\") pod \"e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") " pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:00.719404 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.719291    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-tmp-dir\") pod \"e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") " pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:00.719404 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.719312    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-dshm\") pod \"e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") " pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:00.719404 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.719334    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-model-cache\") pod \"e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") " pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:00.719404 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.719351    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4j6m\" (UniqueName: \"kubernetes.io/projected/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-kube-api-access-r4j6m\") pod \"e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") " pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:00.719813 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.719785    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-tmp-dir\") pod \"e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") " pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:00.719896 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.719798    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-model-cache\") pod \"e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") " pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:00.719896 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.719837    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-home\") pod \"e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") " pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:00.721600 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.721573    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-dshm\") pod \"e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") " pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:00.721771 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.721754    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-tls-certs\") pod \"e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") " pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:00.726813 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.726795    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4j6m\" (UniqueName: \"kubernetes.io/projected/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-kube-api-access-r4j6m\") pod \"e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") " pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:00.866298 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:00.866252    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:01.136237 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:01.136212    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"]
Sep 15 12:05:01.138104 ip-10-0-135-143 kubenswrapper[2562]: W0915 12:05:01.138077    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d340fcb_e49c_45f4_818a_9ffdcc249f8b.slice/crio-149dec0317b74c5d712045b55ab96e53d547401d1cbf20031340e06089873a03 WatchSource:0}: Error finding container 149dec0317b74c5d712045b55ab96e53d547401d1cbf20031340e06089873a03: Status 404 returned error can't find the container with id 149dec0317b74c5d712045b55ab96e53d547401d1cbf20031340e06089873a03
Sep 15 12:05:02.075134 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:02.075099    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t" event={"ID":"1d340fcb-e49c-45f4-818a-9ffdcc249f8b","Type":"ContainerStarted","Data":"99b8ecaae032435e637a1efc6a85cbebd6ef1528f393f55dbb7d7dfe171c3ef6"}
Sep 15 12:05:02.075134 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:02.075135    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t" event={"ID":"1d340fcb-e49c-45f4-818a-9ffdcc249f8b","Type":"ContainerStarted","Data":"149dec0317b74c5d712045b55ab96e53d547401d1cbf20031340e06089873a03"}
Sep 15 12:05:02.093185 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:02.093143    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t" podStartSLOduration=2.0931271909999998 podStartE2EDuration="2.093127191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:05:02.092002136 +0000 UTC m=+2400.754497311" watchObservedRunningTime="2026-09-15 12:05:02.093127191 +0000 UTC m=+2400.755622371"
Sep 15 12:05:02.251738 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:02.251711    2562 scope.go:117] "RemoveContainer" containerID="fd6df5b0764d003529728415b37162d2967dd8ac29c6481bcc0124fbd058e88c"
Sep 15 12:05:02.262399 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:02.262381    2562 scope.go:117] "RemoveContainer" containerID="540ca4fdddcfb44d618c61c5a75fda847eb070c6edaf09d83546450637fedd1d"
Sep 15 12:05:03.053300 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:03.053269    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:05:10.866922 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:10.866888    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:10.867342 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:10.866962    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:10.879740 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:10.879715    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:11.128773 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:11.128688    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:05:19.352943 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:19.352904    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:05:57.412571 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:05:57.412533    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"]
Sep 15 12:06:00.947927 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:00.947894    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:06:02.282237 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:02.282132    2562 scope.go:117] "RemoveContainer" containerID="f27f49baecbcea8b93dae50b638f462532d64598e8d8f27218f4f217741107f8"
Sep 15 12:06:02.289915 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:02.289896    2562 scope.go:117] "RemoveContainer" containerID="dd180783fb78c24cc3fe2aa8fd498e436cf3493e789e05d226b33e4addc5e86b"
Sep 15 12:06:02.296817 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:02.296792    2562 scope.go:117] "RemoveContainer" containerID="910965b72c06842d532f50893af33cb722e224368ad2eee53eaf1b27f12bb5f1"
Sep 15 12:06:10.645697 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:10.645659    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:06:12.431358 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:12.431299    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t" podUID="1d340fcb-e49c-45f4-818a-9ffdcc249f8b" containerName="main" containerID="cri-o://99b8ecaae032435e637a1efc6a85cbebd6ef1528f393f55dbb7d7dfe171c3ef6" gracePeriod=45
Sep 15 12:06:12.683023 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:12.682962    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:06:12.792237 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:12.792208    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-home\") pod \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") "
Sep 15 12:06:12.792442 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:12.792245    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-tls-certs\") pod \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") "
Sep 15 12:06:12.792442 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:12.792287    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-dshm\") pod \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") "
Sep 15 12:06:12.792442 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:12.792377    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-tmp-dir\") pod \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") "
Sep 15 12:06:12.792442 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:12.792416    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4j6m\" (UniqueName: \"kubernetes.io/projected/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-kube-api-access-r4j6m\") pod \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") "
Sep 15 12:06:12.792639 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:12.792442    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-model-cache\") pod \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\" (UID: \"1d340fcb-e49c-45f4-818a-9ffdcc249f8b\") "
Sep 15 12:06:12.792639 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:12.792534    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-home" (OuterVolumeSpecName: "home") pod "1d340fcb-e49c-45f4-818a-9ffdcc249f8b" (UID: "1d340fcb-e49c-45f4-818a-9ffdcc249f8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:06:12.793237 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:12.792675    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d340fcb-e49c-45f4-818a-9ffdcc249f8b" (UID: "1d340fcb-e49c-45f4-818a-9ffdcc249f8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:06:12.793237 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:12.793213    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 12:06:12.796100 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:12.793481    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d340fcb-e49c-45f4-818a-9ffdcc249f8b" (UID: "1d340fcb-e49c-45f4-818a-9ffdcc249f8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:06:12.800042 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:12.800010    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d340fcb-e49c-45f4-818a-9ffdcc249f8b" (UID: "1d340fcb-e49c-45f4-818a-9ffdcc249f8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:06:12.800244 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:12.800175    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-kube-api-access-r4j6m" (OuterVolumeSpecName: "kube-api-access-r4j6m") pod "1d340fcb-e49c-45f4-818a-9ffdcc249f8b" (UID: "1d340fcb-e49c-45f4-818a-9ffdcc249f8b"). InnerVolumeSpecName "kube-api-access-r4j6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:06:12.800334 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:12.800309    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-dshm" (OuterVolumeSpecName: "dshm") pod "1d340fcb-e49c-45f4-818a-9ffdcc249f8b" (UID: "1d340fcb-e49c-45f4-818a-9ffdcc249f8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:06:12.894438 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:12.894402    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 12:06:12.894438 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:12.894435    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 12:06:12.894438 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:12.894446    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 12:06:12.894684 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:12.894480    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 12:06:12.894684 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:12.894491    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4j6m\" (UniqueName: \"kubernetes.io/projected/1d340fcb-e49c-45f4-818a-9ffdcc249f8b-kube-api-access-r4j6m\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Sep 15 12:06:13.383170 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:13.383136    2562 generic.go:356] "Generic (PLEG): container finished" podID="1d340fcb-e49c-45f4-818a-9ffdcc249f8b" containerID="99b8ecaae032435e637a1efc6a85cbebd6ef1528f393f55dbb7d7dfe171c3ef6" exitCode=0
Sep 15 12:06:13.383354 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:13.383211    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"
Sep 15 12:06:13.383354 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:13.383221    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t" event={"ID":"1d340fcb-e49c-45f4-818a-9ffdcc249f8b","Type":"ContainerDied","Data":"99b8ecaae032435e637a1efc6a85cbebd6ef1528f393f55dbb7d7dfe171c3ef6"}
Sep 15 12:06:13.383354 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:13.383266    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t" event={"ID":"1d340fcb-e49c-45f4-818a-9ffdcc249f8b","Type":"ContainerDied","Data":"149dec0317b74c5d712045b55ab96e53d547401d1cbf20031340e06089873a03"}
Sep 15 12:06:13.383354 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:13.383283    2562 scope.go:117] "RemoveContainer" containerID="99b8ecaae032435e637a1efc6a85cbebd6ef1528f393f55dbb7d7dfe171c3ef6"
Sep 15 12:06:13.391506 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:13.391486    2562 scope.go:117] "RemoveContainer" containerID="99b8ecaae032435e637a1efc6a85cbebd6ef1528f393f55dbb7d7dfe171c3ef6"
Sep 15 12:06:13.391784 ip-10-0-135-143 kubenswrapper[2562]: E0915 12:06:13.391762    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b8ecaae032435e637a1efc6a85cbebd6ef1528f393f55dbb7d7dfe171c3ef6\": container with ID starting with 99b8ecaae032435e637a1efc6a85cbebd6ef1528f393f55dbb7d7dfe171c3ef6 not found: ID does not exist" containerID="99b8ecaae032435e637a1efc6a85cbebd6ef1528f393f55dbb7d7dfe171c3ef6"
Sep 15 12:06:13.391879 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:13.391790    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b8ecaae032435e637a1efc6a85cbebd6ef1528f393f55dbb7d7dfe171c3ef6"} err="failed to get container status \"99b8ecaae032435e637a1efc6a85cbebd6ef1528f393f55dbb7d7dfe171c3ef6\": rpc error: code = NotFound desc = could not find container \"99b8ecaae032435e637a1efc6a85cbebd6ef1528f393f55dbb7d7dfe171c3ef6\": container with ID starting with 99b8ecaae032435e637a1efc6a85cbebd6ef1528f393f55dbb7d7dfe171c3ef6 not found: ID does not exist"
Sep 15 12:06:13.407024 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:13.406997    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"]
Sep 15 12:06:13.410413 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:13.410391    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-a80b3a05-kserve-854d8544ff-m7s9t"]
Sep 15 12:06:13.873098 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:13.873044    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d340fcb-e49c-45f4-818a-9ffdcc249f8b" path="/var/lib/kubelet/pods/1d340fcb-e49c-45f4-818a-9ffdcc249f8b/volumes"
Sep 15 12:06:17.050564 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:17.050528    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:06:24.883234 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:24.883185    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2"]
Sep 15 12:06:24.883823 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:24.883643    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d340fcb-e49c-45f4-818a-9ffdcc249f8b" containerName="main"
Sep 15 12:06:24.883823 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:24.883665    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d340fcb-e49c-45f4-818a-9ffdcc249f8b" containerName="main"
Sep 15 12:06:24.883823 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:24.883772    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d340fcb-e49c-45f4-818a-9ffdcc249f8b" containerName="main"
Sep 15 12:06:24.888012 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:24.887991    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2"
Sep 15 12:06:24.891079 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:24.891027    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 15 12:06:24.891213 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:24.891136    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:06:24.891213 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:24.891206    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 12:06:24.891343 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:24.891141    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 12:06:24.891343 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:24.891142    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bjt69\""
Sep 15 12:06:24.898282 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:24.898257    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2"]
Sep 15 12:06:24.950244 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:24.950217    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:06:24.994878 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:24.994850    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414fe5e0-57c1-4616-be80-1947f0fe5745-openshift-service-ca\") pod \"maas-api-64765b4c74-7xmk2\" (UID: \"414fe5e0-57c1-4616-be80-1947f0fe5745\") " pod="odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2"
Sep 15 12:06:24.995034 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:24.994903    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414fe5e0-57c1-4616-be80-1947f0fe5745-metrics-certs\") pod \"maas-api-64765b4c74-7xmk2\" (UID: \"414fe5e0-57c1-4616-be80-1947f0fe5745\") " pod="odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2"
Sep 15 12:06:24.995034 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:24.994987    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2x9h\" (UniqueName: \"kubernetes.io/projected/414fe5e0-57c1-4616-be80-1947f0fe5745-kube-api-access-b2x9h\") pod \"maas-api-64765b4c74-7xmk2\" (UID: \"414fe5e0-57c1-4616-be80-1947f0fe5745\") " pod="odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2"
Sep 15 12:06:24.995034 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:24.995028    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414fe5e0-57c1-4616-be80-1947f0fe5745-maas-api-tls\") pod \"maas-api-64765b4c74-7xmk2\" (UID: \"414fe5e0-57c1-4616-be80-1947f0fe5745\") " pod="odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2"
Sep 15 12:06:25.096171 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:25.096132    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414fe5e0-57c1-4616-be80-1947f0fe5745-metrics-certs\") pod \"maas-api-64765b4c74-7xmk2\" (UID: \"414fe5e0-57c1-4616-be80-1947f0fe5745\") " pod="odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2"
Sep 15 12:06:25.096346 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:25.096198    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2x9h\" (UniqueName: \"kubernetes.io/projected/414fe5e0-57c1-4616-be80-1947f0fe5745-kube-api-access-b2x9h\") pod \"maas-api-64765b4c74-7xmk2\" (UID: \"414fe5e0-57c1-4616-be80-1947f0fe5745\") " pod="odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2"
Sep 15 12:06:25.096346 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:25.096257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414fe5e0-57c1-4616-be80-1947f0fe5745-maas-api-tls\") pod \"maas-api-64765b4c74-7xmk2\" (UID: \"414fe5e0-57c1-4616-be80-1947f0fe5745\") " pod="odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2"
Sep 15 12:06:25.096346 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:25.096329    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414fe5e0-57c1-4616-be80-1947f0fe5745-openshift-service-ca\") pod \"maas-api-64765b4c74-7xmk2\" (UID: \"414fe5e0-57c1-4616-be80-1947f0fe5745\") " pod="odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2"
Sep 15 12:06:25.097110 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:25.097083    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414fe5e0-57c1-4616-be80-1947f0fe5745-openshift-service-ca\") pod \"maas-api-64765b4c74-7xmk2\" (UID: \"414fe5e0-57c1-4616-be80-1947f0fe5745\") " pod="odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2"
Sep 15 12:06:25.098703 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:25.098683    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414fe5e0-57c1-4616-be80-1947f0fe5745-maas-api-tls\") pod \"maas-api-64765b4c74-7xmk2\" (UID: \"414fe5e0-57c1-4616-be80-1947f0fe5745\") " pod="odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2"
Sep 15 12:06:25.098967 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:25.098947    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414fe5e0-57c1-4616-be80-1947f0fe5745-metrics-certs\") pod \"maas-api-64765b4c74-7xmk2\" (UID: \"414fe5e0-57c1-4616-be80-1947f0fe5745\") " pod="odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2"
Sep 15 12:06:25.104134 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:25.104113    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2x9h\" (UniqueName: \"kubernetes.io/projected/414fe5e0-57c1-4616-be80-1947f0fe5745-kube-api-access-b2x9h\") pod \"maas-api-64765b4c74-7xmk2\" (UID: \"414fe5e0-57c1-4616-be80-1947f0fe5745\") " pod="odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2"
Sep 15 12:06:25.198409 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:25.198336    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2"
Sep 15 12:06:25.472746 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:25.472687    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2"]
Sep 15 12:06:25.474998 ip-10-0-135-143 kubenswrapper[2562]: W0915 12:06:25.474970    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414fe5e0_57c1_4616_be80_1947f0fe5745.slice/crio-bad722a5adb0bf01345b3b3c6400fa93ec09f7f34e6a16f61921303e2cee0211 WatchSource:0}: Error finding container bad722a5adb0bf01345b3b3c6400fa93ec09f7f34e6a16f61921303e2cee0211: Status 404 returned error can't find the container with id bad722a5adb0bf01345b3b3c6400fa93ec09f7f34e6a16f61921303e2cee0211
Sep 15 12:06:25.476236 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:25.476218    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 12:06:26.449017 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:26.448981    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2" event={"ID":"414fe5e0-57c1-4616-be80-1947f0fe5745","Type":"ContainerStarted","Data":"bad722a5adb0bf01345b3b3c6400fa93ec09f7f34e6a16f61921303e2cee0211"}
Sep 15 12:06:28.461189 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:28.461151    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2" event={"ID":"414fe5e0-57c1-4616-be80-1947f0fe5745","Type":"ContainerStarted","Data":"9cf7459c01fd87cd33f86a81841587f7e91abfdae97bc31e10494239a7332651"}
Sep 15 12:06:28.461542 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:28.461275    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2"
Sep 15 12:06:28.479590 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:28.479507    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2" podStartSLOduration=1.7344313150000001 podStartE2EDuration="4.479494408s" totalImagesPullingTime="2.745063093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:06:28.478159307 +0000 UTC m=+2487.140654512" watchObservedRunningTime="2026-09-15 12:06:28.479494408 +0000 UTC m=+2487.141989590"
Sep 15 12:06:32.044853 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:32.044818    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:06:34.471977 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:34.471947    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-64765b4c74-7xmk2"
Sep 15 12:06:40.442575 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:40.442532    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:06:47.846382 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:47.846344    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mgpww"]
Sep 15 12:06:56.475116 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:56.475007    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-strq5_fc794b58-48a4-4697-a523-7e46fcf3464a/manager/0.log"
Sep 15 12:06:57.171113 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:57.171087    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b96c7c888-d4f46_cb818920-9a4c-4a0e-8899-7d62c5c3601d/copy-manifests/0.log"
Sep 15 12:06:57.184455 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:57.184418    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b96c7c888-d4f46_cb818920-9a4c-4a0e-8899-7d62c5c3601d/manager/0.log"
Sep 15 12:06:57.620255 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:57.620222    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-64765b4c74-7xmk2_414fe5e0-57c1-4616-be80-1947f0fe5745/maas-api/0.log"
Sep 15 12:06:57.939278 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:57.939201    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29824560-k4v2h_94a51cee-66e1-4e3c-ae56-3223ab47c915/cleanup/0.log"
Sep 15 12:06:59.129363 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:59.129334    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xdz2w_beb108cb-5872-4cb9-8733-9dceae10f816/kuadrant-console-plugin/0.log"
Sep 15 12:06:59.360738 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:59.360710    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-mgpww_907a5c3b-ceb5-43d3-96c4-c4ba68f8f613/limitador/0.log"
Sep 15 12:06:59.481784 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:06:59.481721    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-qdr85_e43941e0-3e20-4744-98b1-55855559ca7e/manager/0.log"
Sep 15 12:07:00.903564 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:00.903529    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-qt6tr_8256e21e-62b3-4003-b940-21ba1ca1e0bc/main/0.log"
Sep 15 12:07:01.117831 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:01.117803    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-2t5jb_1781d3ca-d649-4849-b55a-a1bbc9b75f91/main/0.log"
Sep 15 12:07:01.548555 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:01.548524    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-7spp4_f43c5521-3809-429e-9ce0-3dd0c8ef5299/main/0.log"
Sep 15 12:07:02.318942 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:02.318910    2562 scope.go:117] "RemoveContainer" containerID="f289211ca8a010eb8a3ec2be0abdb95df7c97859fe74d2ef1030e6a4d8f0362c"
Sep 15 12:07:13.007962 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:13.007926    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7j9vh_85b6a362-7f90-4e54-89d2-2c21ef687efb/global-pull-secret-syncer/0.log"
Sep 15 12:07:13.134809 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:13.134776    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sr9gc_870be114-239c-4f62-8def-c3144e7cbeb0/konnectivity-agent/0.log"
Sep 15 12:07:13.206944 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:13.206916    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-143.ec2.internal_4fc673a9ff7f802e0d691cff004f3253/haproxy/0.log"
Sep 15 12:07:18.070699 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:18.070667    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xdz2w_beb108cb-5872-4cb9-8733-9dceae10f816/kuadrant-console-plugin/0.log"
Sep 15 12:07:18.159828 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:18.159803    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-mgpww_907a5c3b-ceb5-43d3-96c4-c4ba68f8f613/limitador/0.log"
Sep 15 12:07:18.237232 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:18.237205    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-qdr85_e43941e0-3e20-4744-98b1-55855559ca7e/manager/0.log"
Sep 15 12:07:20.435404 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:20.435372    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qtn7k_9fbb571d-8b56-4afb-ba33-04283dc19f69/node-exporter/0.log"
Sep 15 12:07:20.455890 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:20.455843    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qtn7k_9fbb571d-8b56-4afb-ba33-04283dc19f69/kube-rbac-proxy/0.log"
Sep 15 12:07:20.480991 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:20.480964    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qtn7k_9fbb571d-8b56-4afb-ba33-04283dc19f69/init-textfile/0.log"
Sep 15 12:07:22.581364 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:22.581337    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-p6cx4_b085c684-f385-4e33-b780-492f1aac71b8/networking-console-plugin/0.log"
Sep 15 12:07:23.736252 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.736220    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"]
Sep 15 12:07:23.739454 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.739423    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"
Sep 15 12:07:23.741583 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.741559    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nkxzt\"/\"openshift-service-ca.crt\""
Sep 15 12:07:23.741703 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.741630    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nkxzt\"/\"kube-root-ca.crt\""
Sep 15 12:07:23.741703 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.741690    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nkxzt\"/\"default-dockercfg-jpntf\""
Sep 15 12:07:23.746830 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.746810    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"]
Sep 15 12:07:23.772558 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.772524    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc1ea257-f065-43cb-8928-5134d5339ef2-lib-modules\") pod \"perf-node-gather-daemonset-w4cnp\" (UID: \"fc1ea257-f065-43cb-8928-5134d5339ef2\") " pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"
Sep 15 12:07:23.772682 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.772578    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fc1ea257-f065-43cb-8928-5134d5339ef2-podres\") pod \"perf-node-gather-daemonset-w4cnp\" (UID: \"fc1ea257-f065-43cb-8928-5134d5339ef2\") " pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"
Sep 15 12:07:23.772682 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.772603    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc1ea257-f065-43cb-8928-5134d5339ef2-sys\") pod \"perf-node-gather-daemonset-w4cnp\" (UID: \"fc1ea257-f065-43cb-8928-5134d5339ef2\") " pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"
Sep 15 12:07:23.772682 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.772624    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fc1ea257-f065-43cb-8928-5134d5339ef2-proc\") pod \"perf-node-gather-daemonset-w4cnp\" (UID: \"fc1ea257-f065-43cb-8928-5134d5339ef2\") " pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"
Sep 15 12:07:23.772796 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.772702    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qj99\" (UniqueName: \"kubernetes.io/projected/fc1ea257-f065-43cb-8928-5134d5339ef2-kube-api-access-9qj99\") pod \"perf-node-gather-daemonset-w4cnp\" (UID: \"fc1ea257-f065-43cb-8928-5134d5339ef2\") " pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"
Sep 15 12:07:23.873157 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.873132    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qj99\" (UniqueName: \"kubernetes.io/projected/fc1ea257-f065-43cb-8928-5134d5339ef2-kube-api-access-9qj99\") pod \"perf-node-gather-daemonset-w4cnp\" (UID: \"fc1ea257-f065-43cb-8928-5134d5339ef2\") " pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"
Sep 15 12:07:23.873327 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.873179    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc1ea257-f065-43cb-8928-5134d5339ef2-lib-modules\") pod \"perf-node-gather-daemonset-w4cnp\" (UID: \"fc1ea257-f065-43cb-8928-5134d5339ef2\") " pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"
Sep 15 12:07:23.873327 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.873213    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fc1ea257-f065-43cb-8928-5134d5339ef2-podres\") pod \"perf-node-gather-daemonset-w4cnp\" (UID: \"fc1ea257-f065-43cb-8928-5134d5339ef2\") " pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"
Sep 15 12:07:23.873327 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.873234    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc1ea257-f065-43cb-8928-5134d5339ef2-sys\") pod \"perf-node-gather-daemonset-w4cnp\" (UID: \"fc1ea257-f065-43cb-8928-5134d5339ef2\") " pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"
Sep 15 12:07:23.873327 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.873257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fc1ea257-f065-43cb-8928-5134d5339ef2-proc\") pod \"perf-node-gather-daemonset-w4cnp\" (UID: \"fc1ea257-f065-43cb-8928-5134d5339ef2\") " pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"
Sep 15 12:07:23.873490 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.873339    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc1ea257-f065-43cb-8928-5134d5339ef2-sys\") pod \"perf-node-gather-daemonset-w4cnp\" (UID: \"fc1ea257-f065-43cb-8928-5134d5339ef2\") " pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"
Sep 15 12:07:23.873490 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.873358    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fc1ea257-f065-43cb-8928-5134d5339ef2-proc\") pod \"perf-node-gather-daemonset-w4cnp\" (UID: \"fc1ea257-f065-43cb-8928-5134d5339ef2\") " pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"
Sep 15 12:07:23.873490 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.873360    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc1ea257-f065-43cb-8928-5134d5339ef2-lib-modules\") pod \"perf-node-gather-daemonset-w4cnp\" (UID: \"fc1ea257-f065-43cb-8928-5134d5339ef2\") " pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"
Sep 15 12:07:23.873490 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.873362    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fc1ea257-f065-43cb-8928-5134d5339ef2-podres\") pod \"perf-node-gather-daemonset-w4cnp\" (UID: \"fc1ea257-f065-43cb-8928-5134d5339ef2\") " pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"
Sep 15 12:07:23.881707 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:23.881685    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qj99\" (UniqueName: \"kubernetes.io/projected/fc1ea257-f065-43cb-8928-5134d5339ef2-kube-api-access-9qj99\") pod \"perf-node-gather-daemonset-w4cnp\" (UID: \"fc1ea257-f065-43cb-8928-5134d5339ef2\") " pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"
Sep 15 12:07:24.050215 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:24.050141    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"
Sep 15 12:07:24.322227 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:24.322132    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"]
Sep 15 12:07:24.324841 ip-10-0-135-143 kubenswrapper[2562]: W0915 12:07:24.324809    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfc1ea257_f065_43cb_8928_5134d5339ef2.slice/crio-cbd77ceeb2ea4dfdbfed07dde3b0cd462c52b738dec00cee76cd8eadc82add3c WatchSource:0}: Error finding container cbd77ceeb2ea4dfdbfed07dde3b0cd462c52b738dec00cee76cd8eadc82add3c: Status 404 returned error can't find the container with id cbd77ceeb2ea4dfdbfed07dde3b0cd462c52b738dec00cee76cd8eadc82add3c
Sep 15 12:07:24.703929 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:24.703890    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp" event={"ID":"fc1ea257-f065-43cb-8928-5134d5339ef2","Type":"ContainerStarted","Data":"12acc1b4830a27b1a0ddfd816fc9c0b48cc9adadf73875794f51c128adf4f140"}
Sep 15 12:07:24.704114 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:24.703946    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp" event={"ID":"fc1ea257-f065-43cb-8928-5134d5339ef2","Type":"ContainerStarted","Data":"cbd77ceeb2ea4dfdbfed07dde3b0cd462c52b738dec00cee76cd8eadc82add3c"}
Sep 15 12:07:24.704114 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:24.703988    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"
Sep 15 12:07:24.720459 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:24.720413    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp" podStartSLOduration=1.720399332 podStartE2EDuration="1.720399332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:07:24.718825301 +0000 UTC m=+2543.381320494" watchObservedRunningTime="2026-09-15 12:07:24.720399332 +0000 UTC m=+2543.382894521"
Sep 15 12:07:25.773428 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:25.773398    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9wtp8_8b887a34-bd19-4fa3-9897-a03605d5231c/dns/0.log"
Sep 15 12:07:25.793113 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:25.793088    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9wtp8_8b887a34-bd19-4fa3-9897-a03605d5231c/kube-rbac-proxy/0.log"
Sep 15 12:07:25.931309 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:25.931274    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f5l6r_dbb4a488-5113-44e5-bea0-b831896d7cac/dns-node-resolver/0.log"
Sep 15 12:07:26.479111 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:26.479078    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-759494458-fl68h_0fb1ee35-a207-42fa-b165-dfa8c7e93a86/registry/0.log"
Sep 15 12:07:26.497114 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:26.497095    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-42tpl_85bb9ddc-8249-4923-a96d-4b17b5450c51/node-ca/0.log"
Sep 15 12:07:29.409043 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:29.409009    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tgsbr_356a69de-a242-4342-87a5-f876c681dd1e/serve-healthcheck-canary/0.log"
Sep 15 12:07:29.978248 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:29.978205    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rs74z_27c60b28-4026-4737-a4f2-b69c08522dae/kube-rbac-proxy/0.log"
Sep 15 12:07:29.998424 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:29.998396    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rs74z_27c60b28-4026-4737-a4f2-b69c08522dae/exporter/0.log"
Sep 15 12:07:30.017974 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:30.017953    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rs74z_27c60b28-4026-4737-a4f2-b69c08522dae/extractor/0.log"
Sep 15 12:07:30.717627 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:30.717599    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nkxzt/perf-node-gather-daemonset-w4cnp"
Sep 15 12:07:32.212522 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:32.212494    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-strq5_fc794b58-48a4-4697-a523-7e46fcf3464a/manager/0.log"
Sep 15 12:07:33.034835 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:33.034754    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b96c7c888-d4f46_cb818920-9a4c-4a0e-8899-7d62c5c3601d/manager/0.log"
Sep 15 12:07:33.054462 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:33.054438    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b96c7c888-d4f46_cb818920-9a4c-4a0e-8899-7d62c5c3601d/copy-manifests/0.log"
Sep 15 12:07:34.505573 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:34.505535    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-667dbb5fb5-ngctn_3ed95d68-12dc-4162-a244-6113aeff38ba/manager/0.log"
Sep 15 12:07:40.144252 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:40.144223    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-tlpgr_daf3093b-fc68-418c-ac0c-ccc1af1f007b/migrator/0.log"
Sep 15 12:07:40.168771 ip-10-0-135-143 kubenswrapper[2562]: I0915 12:07:40.168744    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-tlpgr_daf3093b-fc68-418c-ac0c-ccc1af1f007b/graceful-termination/0.log"